Elamur Research⌕
☰
home / papers / 2019 / CODAH: An Adversarially-Authored Question Answering Dataset for Common Sense

CODAH: An Adversarially-Authored Question Answering Dataset for Common Sense

Paper · 2019 · Michael Chen, Mike D'Arcy, Alisa Liu et al.

Summary

Pre-trained transformer models such as BERT reached or exceeded human scores on SWAG, a sentence-completion benchmark for commonsense inference. The authors argue that these models lack real human-like common sense and instead exploit regularities in that dataset. To address this, they build CODAH, a harder companion set in the same format: a prompt sentence, a subject, and four candidate endings, of which exactly one is sensible. Most questions were written by Northwestern computer science students who could see, in real time, how a BERT-base model fine-tuned on SWAG answered them. Students earned extra credit when a question fooled this model, and equal credit when it still fooled models fine-tuned on other submissions in cross-validation, which discouraged easily learnable artifacts. After manual cleaning, 2,801 questions remain. Even models fine-tuned on SWAG and then on CODAH reach only about 65% accuracy, while humans score 95.3%. The work offers a practical recipe for model-in-the-loop adversarial authoring and a tougher commonsense test. The authors note, however, that CODAH complements SWAG rather than being harder at equal size.

Key points

  • Dataset construction: 4,149 raw submissions were reduced to 2,801 questions from 116 named contributors, with 321 (11%) submitted anonymously. The data is released at github.com/Websail-NU/CODAH.
  • Main results: In cross-validation, OpenAI GPT-1 scored 65.3% and BERT scored 64.5%, both after prior fine-tuning on SWAG. Human accuracy was 95.3%, compared with about 85% for humans on SWAG.
  • Question categories: Questions were labeled as Idioms, Negation, Polysemy, Reference, and Quantitative Reasoning. Quantitative questions were especially hard for both BERT and GPT.
  • Answer-only baseline: With the question hidden, BERT reached 28.4% and GPT-1 reached 53.9%. This indicates some signal lies in the answer options alone.
  • Size-matched comparison: On a SWAG subset equal in size to CODAH, GPT scored 63.6% versus 62.4% on CODAH, and BERT scored 28.7% versus 49.6%. CODAH is therefore distinct from SWAG but not necessarily more difficult at equal size.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Northwestern UniversityEvanston · United States12 works · 1,971 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 510 worksQuestion answering · 242 works

Cite

Cite as Michael Chen, Mike D'Arcy, Alisa Liu, Jared Fernandez and Doug Downey. 2019. CODAH: An Adversarially-Authored Question Answering Dataset for Common Sense. Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-2008

@inproceedings{chen2019codah,
  title = {{CODAH: An Adversarially-Authored Question Answering Dataset for Common Sense}},
  author = {Michael Chen, Mike D'Arcy, Alisa Liu, Jared Fernandez and Doug Downey},
  booktitle = {Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP},
  year = {2019},
  doi = {10.18653/v1/w19-2008},
  url = {https://doi.org/10.18653/v1/w19-2008}
}

References

9 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392.
  2. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642. find
  3. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008.
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  5. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… find
  6. Allyson Ettinger, Sudha Rao, Hal Daumé III, and Emily M Bender. 2017. Towards linguistically generalizable nlp systems: A workshop and shared task. In Proceedings of the First Workshop on Building Linguistically Generalizable NLP Systems, pages 1– 10. find
  7. Reza Ghaeini, Sadid A Hasan, Vivek Datla, Joey Liu, Kathy Lee, Ashequl Qadir, Yuan Ling, Aaditya Prakash, Xiaoli Fern, and Oladimeji Farri. 2018. Dr-bilstm: Dependent reading bidirectional lstm for natural language inference. In Proceedings of the 2018 Conference of the North American Chapter of the… find
  8. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. Swag: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93–104. find
  9. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2019-codah-adversarially-authored-question-answering/