Elamur Research⌕
☰
home / papers / 2026 / The AI Committee: A Multi-Agent Framework for Automated Validation and Remediation of Web-Sourced Data

The AI Committee: A Multi-Agent Framework for Automated Validation and Remediation of Web-Sourced Data

Paper · 2026 · Sunith Vallabhaneni, Thomas Berkane and Maimuna S. Majumder

Summary

LLM-based web agents can now build research datasets from web pages, but their output often contains invented or missing values, misread page content, and invalid entries that are slow to catch by hand. The authors present the AI Committee (AIC), an open-source, model-agnostic multi-agent system that checks and repairs tabular web-sourced data without fine-tuning. The user supplies a dataset with a source URL per row, a short description of its purpose, and the fields to check. AIC first derives dataset-specific rules and negative examples from a small sample through in-context learning. It then routes each record through specialised agents: relevancy screening, source scrutiny, chain-of-thought fact-checking, and integrity and format checks. A remediation loop fixes rejected rows and pulls new records from the source pages. On three LLM-collected datasets with human-cleaned ground truth, AIC outperformed a single-prompt monolithic agent and a rule-based cleaner. The cheapest backbone tested, gpt-4o-mini, gave the best F1 at a fraction of the cost, which suits research groups with limited budgets.

Key points

  • Evaluation data: three web-collected datasets:

    • Natural Disasters: 125 points on Haiti and Cameroon.
    • Police Misconduct: 95 points from the US.
    • COVID-19 Tracing: 83 points on contact-tracing app downloads per US state.

    Three public health researchers annotated the ground truth under an IAA protocol.

  • Pipeline components: a ContextGenerator, a Relevancy Assessor, source scrutiny, a FactCheckerAgent, an ArbiterAgent, and a DataRemediationAgent.

    • The ContextGenerator infers each field's semantics and negative examples from 10 random rows.
    • The FactCheckerAgent combines CoT reasoning with a semantic audit.
    • The ArbiterAgent passes reasons for rejection to the DataRemediationAgent, which corrects the records.
    • A schema generated from the user's field list enforces structure.

  • Main results: AIC with gpt-4o-mini reached F1 85.1 (precision 92.7, recall 78.7) at about $0.30 per batch.

    • The gpt-5 monolithic baseline scored F1 71.0 at $4.95, making AIC more than 16× cheaper.
    • The rule-based baseline scored F1 71.4 and had the lowest recall (58.2).
    • AIC with gpt-5 reached 100% precision but with lower recall and much higher cost and latency.

  • Ablation: every agent contributes to performance.

    • Removing the formatter caused the largest F1 drop (−28.2).
    • Removing relevancy screening cost −12.0 F1, and removing remediation cost −6.4.

  • Availability: code, the three annotated datasets, and the IAA protocol are released under Apache 2.0 on GitHub (sunith-v/theAICommittee).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Harvard UniversityCambridge · United States112 works · 42,023 citations🏥Boston Children's HospitalBoston · United States6 works · 678 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 worksDecision Sciences · 209 worksManagement Science and Operations Research · 108 works

Cite

Cite as Sunith Vallabhaneni, Thomas Berkane and Maimuna S. Majumder. 2026. The AI Committee: A Multi-Agent Framework for Automated Validation and Remediation of Web-Sourced Data. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 3: System Demonstrations).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-demo.41

@inproceedings{vallabhaneni2026ai,
  title = {{The AI Committee: A Multi-Agent Framework for Automated Validation and Remediation of Web-Sourced Data}},
  author = {Sunith Vallabhaneni, Thomas Berkane and Maimuna S. Majumder},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 3: System Demonstrations)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-demo.41},
  url = {https://doi.org/10.18653/v1/2026.eacl-demo.41}
}

References

8 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tommaso Bendinelli, Artur Dox, and Christian Holz. 2025. Exploring LLM agents for cleaning tabular machine learning datasets. arXiv preprint arXiv:2503.06664. Proceedings of the ICLR 2025 Workshop on Foundation Models in the Wild. — [arXiv:2503.06664](https://arxiv.org/abs/2503.06664) find
  2. Thomas Berkane, Marie-Laure Charpignon, and Maimuna S. Majumder. 2025. LLM-based web data collection for research dataset creation. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 12610–12622. Association for Computational Linguistics. find
  3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, and 1 others. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901.
  4. Wenhao Huang, Zhouhong Gu, Chenghao Peng, Zhixu Li, Jiaqing Liang, Yanghua Xiao, Liqian Wen, and Zulong Chen. 2024. AutoScraper: A progressive understanding web agent for web scraper generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2371–238… find
  5. Tianyi Ma, Yiyue Qian, Zheyuan Zhang, Zehong Wang, Xiaoye Qian, Feifan Bai, Yifan Ding, Xuwei Luo, Shinan Zhang, Keerthiram Murugesan, Chuxu Zhang, and Yanfang Ye. 2025. Auto-Data: A multi-agent system for open web data collection. arXiv preprint arXiv:2505.15859. — [arXiv:2505.15859](https://arxiv.org/abs/2505.15859) find
  6. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with… find
  7. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824– 24837.
  8. Shuo Zhang, Zezhou Huang, and Eugene Wu. 2024. Data cleaning using large language models. arXiv preprint arXiv:2410.15547. — [arXiv:2410.15547](https://arxiv.org/abs/2410.15547) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vallabhaneni-2026-ai-committee-multi-agent-framework/