When building machine learning models—especially in domains like mental health, social behavior, or language—everything depends on the quality of your labels. But most annotation systems assume labels are correct as soon as someone clicks an option. But what if disagreement itself was valuable? What if instead of hiding uncertainty, you could measure it? That’s where JuryDuty comes in. JuryDuty is a data annotation platform built around the idea that labels should be earned, not assumed. Instead of relying on a single annotator, the same data sample is intentionally shown to multiple annotators. But how does this help? With JuryDuty, every annotation is stored independently, allowing the system to explicitly measure inter-annotator agreement using statistical metrics like Cohen’s Kappa. This makes it easy to identify samples where annotators disagree, instead of silently averaging them or treating them as ground truth. So how does it work? Researchers define labeling tasks and upload data samples to the platform. Each sample is assigned to multiple annotators, who work independently. Once annotations are complete, JuryDuty computes agreement scores, highlights low-confidence samples, and makes disagreement visible for re-annotation or expert review. The data model cleanly separates tasks, annotators, and annotations, making the entire process auditable and reproducible. The result is a research-grade labeling pipeline that prioritizes label reliability over speed, making it suitable for sensitive or high-stakes datasets. Tech Stack: • Frontend: SvelteKit • Backend: Supabase (PostgreSQL, Auth) JuryDuty – Because labels shouldn’t be guesses.