Elamur Research⌕
☰
home / papers / 2026 / RE-AD: Real-Time Requirement Adherence for Data Labeling

RE-AD: Real-Time Requirement Adherence for Data Labeling

Paper · 2026 · Siddarth Malreddy, Ishan Nigam, Akshay Arora et al.

Summary

Crowd-sourced annotations used to train LLMs often violate the Standard Operating Procedures (SOPs) that define a labeling task. Conventional QA, such as inter-annotator agreement or expert spot-checks, catches these errors only after collection ends, which forces costly rework. RE-AD, developed at Uber AI Solutions, moves checking into the annotation tool so it happens while annotators work.

The framework has two stages:

  • Offline atomization. An LLM agent converts SOP documents to Markdown and extracts rules iteratively. A reflection step tests each rule for atomicity and non-overlap. Rules are then sorted into formatting, simple and subjective tiers.
  • Online validation. All rules are checked in parallel. Formatting rules use Python code, simple rules use Gemini 3 Flash zero-shot, and subjective rules use Gemini 3 Pro with CoT. Prefix caching keeps latency low.

On RE-AD-Eval, a synthetic benchmark with injected violations, the system reaches a total F1 of 0.749 at a median latency of about 2.3 s. In production, annotators accepted and corrected 82% of flagged errors. This suggests real-time guideline checking is practical, although subjective rules remain difficult.

Key points

  • RE-AD-Eval benchmark. It is built around a 20-rule SOP with 8 formatting, 5 simple and 7 subjective rules. Three annotator personas (Expert, Intermediate, Novice) inject errors at tier-specific rates of 5–25%, 8–35% and 12–50%. The data was synthesized with OpenAI models and validated with Gemini to avoid self-bias.
  • Per-tier F1. RE-AD scores 1.000 on formatting, 0.864 on simple rules and only 0.551 on subjective rules.
  • Ablation against monolithic batch prompting.
    • Batch with Gemini 3 Pro reaches a total F1 of 0.742 but needs about 36.9 s, over 16× slower than RE-AD.
    • Batch with Gemini 3 Flash takes 5.8 s and reaches 0.628.
    • Batch is weak on formatting (F1 as low as 0.471) but beats RE-AD on subjective rules (0.612).
  • Stability. Rule atomization varies by less than 5% across repeated runs on the same SOP.
  • Limitations.
    • Only English tasks were tested.
    • The 82% figure is an operational estimate that may be confounded by UI changes and annotator learning.
    • Subjective-tier verdicts are treated as advisory rather than as gates.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Science Applications · 131 works

Cite

Cite as Siddarth Malreddy, Ishan Nigam, Akshay Arora, Nikhil Mittal and Subrat Sahu. 2026. RE-AD: Real-Time Requirement Adherence for Data Labeling. Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.gem-main.17

@inproceedings{malreddy2026re,
  title = {{RE-AD: Real-Time Requirement Adherence for Data Labeling}},
  author = {Siddarth Malreddy, Ishan Nigam, Akshay Arora, Nikhil Mittal and Subrat Sahu},
  booktitle = {Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)},
  year = {2026},
  doi = {10.18653/v1/2026.gem-main.17},
  url = {https://doi.org/10.18653/v1/2026.gem-main.17}
}

References

18 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596. find
  2. Fabrizio Gilardi, Meysam Alizadeh, and Maoel Kubli. 2023. ChatGPT outperforms crowd-workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30).
  3. Jiawei Gu, Xuhui Ye, Zhichao Gao, Qingyu Feng, Da Yu, Liang Wang, Xu Dai, and 1 others. 2024. A survey on LLM-as-a-judge. arXiv preprint arXiv:2411.15594. find
  4. Xingwei He, Zhenghao Lin, Yeyun Gong, Alex Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2024. AnnoLLM: Making large language models to be better crowdsourced annotators. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Compu…
  5. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2023. Towards mitigating LLM hallucination via self reflection. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1827–1843.
  6. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.
  7. Ximing Lu, Sean Welleck, Peter West, Liwei Jiang, Jungo Kasai, Daniel Khashabi, Ronan Le Bras, Hao Lian, Lianhui Yu, Maarten Sap, and 1 others. 2020. Neulogic: A neural-symbolic model for structured prediction. arXiv preprint arXiv:2010.04306. — [arXiv:2010.04306](https://arxiv.org/abs/2010.04306) find
  8. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, and 1 others. 2023. Self-refine: Iterative refinement with self-feedback. In Advances in Neural Information Processing Systems, volume 36, pages 46534– 46594. find
  9. James Pustejovsky and Amber Stubbs. 2012. Natural Language Annotation for Machine Learning. O’Reilly Media. find
  10. Burr Settles. 2009. Active learning literature survey. Computer Sciences Technical Report 1648, University of Wisconsin–Madison.
  11. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast — but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 254–263.
  12. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Narang Sharan, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations.
  13. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837.
  14. Zhen Xie, Yao Li, Yuxiang Zhang, Minglai Gao, Wenwen Zhang, Rui Zhang, Weiran Chen, and Qing Liu. 2024. Large language models for data annotation and synthesis: A survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 927–958.
  15. Xinghua Zhang, Bowen Yu, Haiyang Yu, Yangyu Lv, Tingwen Liu, Fei Huang, Hongbo Xu, and Yongbin Li. 2023. Wider and deeper LLM networks are fairer LLM evaluators. arXiv preprint arXiv:2308.01862.
  16. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 36, pages 46595–46623.
  17. Eduardo Mosqueira-Rey, Elena Hernandez-Pereira, David Alonso-Rios, José Bobes-Bascán, and Angeles Fernandez-Leal. 2023. Human-in-the-loop machine learning: a state of the art. Artificial Intelligence Review, 56(4):3005–3054. find
  18. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pag…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/malreddy-2026-re-ad-real-time-requirement/