Elamur Research⌕
☰
home / papers / 2021 / General-Purpose Question-Answering with Macaw

General-Purpose Question-Answering with Macaw

Paper · 2021 · Oyvind Tafjord and Peter Clark

Summary

Few strong, general-purpose question-answering systems were openly available in 2021. GPT-3 was closed, UnifiedQA mostly selected spans or multiple-choice options rather than writing free answers, and T5-CBQA lagged in the authors' tests. Tafjord and Clark address this gap with Macaw, a generative QA model built on UnifiedQA, which itself builds on T5. Macaw is trained "multi-angle": each example is split into slots (context, question, MC options, answer, explanation), and the model learns to predict any chosen subset of slots from others. This lets one model answer questions, write questions for a given answer, propose distractor options, or produce short explanations. On Challenge300, a hand-built set of 300 difficult probing questions, the 11B Macaw answers zero-shot more than 10 points (absolute) better than GPT-3, Jurassic-1, and T5-CBQA, despite being roughly sixteen times smaller than GPT-3. The authors also catalogue its failure modes, including false presuppositions, commonsense slips, and repetitive iterative outputs. The open release gives researchers a free baseline and a starting point for fine-tuning.

Key points

  • Inputs and outputs are encoded as plain text slot strings (e.g., $answer$ ; $question$ = …). Angles such as QMC→AE, QC→A, QAC→M and AC→QM are sampled during training, and slot order is shuffled, except that context is placed last because of T5's 512-token limit.
  • Training data include multiple-choice datasets such as ARC and RACE. A Macaw-based model was reported as leading the ARC leaderboards.
  • On Challenge300, Macaw (11B) beats GPT-3 (175B) by over 10% absolute. T5-CBQA (trained on Natural Questions) and AI21's Jurassic-1 also score lower.
  • Known weaknesses include being fooled by false presuppositions (e.g., Zuckerberg "founding Google"), giving overly brief answers, and repeating itself in iterative or story-generation prompts. Like other pretrained models, it can produce biased or offensive answers.
  • Code and model are available at github.com/allenai/macaw, together with all compared models' Challenge300 answers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksComputer Vision and Pattern Recognition · 3,621 works

🧭 Topics 1Research subjects and languages the work is about.Question answering · 491 works

Cite

Cite as Oyvind Tafjord and Peter Clark. 2021. General-Purpose Question-Answering with Macaw. arXiv:2109.02593.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2109.02593 · arXiv 2109.02593

@misc{tafjord2021general,
  title = {{General-Purpose Question-Answering with Macaw}},
  author = {Oyvind Tafjord and Peter Clark},
  year = {2021},
  doi = {10.48550/arxiv.2109.02593},
  eprint = {2109.02593},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2109.02593}
}

References

22 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. Bhakthavatsalam, D. Khashabi, T. Khot, B. D. Mishra, K. Richardson, A. Sabharwal, C. Schoenick, O. Tafjord, and P. Clark. Think you have solved direct-answer question answering? try arc-da, the direct-answer ai2 reasoning challenge. ArXiv, abs/2102.03315, 2021. — [arXiv:2102.03315](https://arxiv.org/abs/2102.03315) find
  2. T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research. Transactions of th…
  3. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner… — Language Models are Few-Shot Learners
  4. G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy. RACE: Large-scale reading comprehension dataset from examinations. In EMNLP, 2017.
  5. H. Levesque, E. Davis, and L. Morgenstern. The winograd schema challenge. In KR, 2011. find
  6. P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? Try ARC, the AI2 Reasoning Challenge. ArXiv, abs/1803.05457, 2018.
  7. T. Li, D. Khashabi, T. Khot, A. Sabharwal, and V. Srikumar. Unqovering stereotyping biases via underspecified questions. In EMNLP, 2020. G. Marcus and E. Davis. Experiments testing gpt-3’s ability at commonsense reasoning: Results. Technical report, NYU, 2020a. (https://cs.nyu.edu/∼davise/papers/GPT…
  8. P. Clark, O. Etzioni, T. Khot, A. Sabharwal, O. Tafjord, P. D. Turney, and D. Khashabi. Combining retrieval, statistics, and inference to answer elementary science questions. In AAAI, 2016. find
  9. G. Marcus and E. Davis. Gpt-3, bloviator: Openai’s language generator has no idea what it’s talking about. MIT Technology Review, Aug 2020b. find
  10. B. Dalvi, L. Huang, N. Tandon, W. tau Yih, and P. Clark. Tracking state changes in procedural text: a challenge dataset and models for process paragraph comprehension. In NAACL-HLT, 2018. find
  11. A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. Improving language understanding by generative pre-training. Technical report, OpenAI, 2018. find
  12. E. Davis. How to write science questions that are easy for people and hard for computers. AI Mag., 37:13–22, 2016. find
  13. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.
  14. P. Hase, S. Zhang, H. Xie, and M. Bansal. Leakage-adjusted simulatability: Can models generate non-trivial explanations of their behavior in natural language? In EMNLP, 2020.
  15. A. Roberts, C. Raffel, and N. M. Shazeer. How much knowledge can you pack into the parameters of a language model? In EMNLP, 2020. find
  16. M. Henaff, J. Weston, A. D. Szlam, A. Bordes, and Y. Le-Cun. Tracking the world state with recurrent entity networks. In ICLR, 2016.
  17. N. Tandon, B. Dalvi, J. Grus, W. tau Yih, A. Bosselut, and P. Clark. Reasoning about actions and state changes by injecting commonsense knowledge. In EMNLP, 2018. find
  18. A. Holtzman, J. Buys, M. Forbes, and Y. Choi. The curious case of neural text degeneration. ArXiv, abs/1904.09751, 2020.
  19. J. Weston, A. Bordes, S. Chopra, and T. Mikolov. Towards AI-Complete question answering: A set of prerequisite toy tasks. In ICLR, 2016.
  20. P. A. Jansen, E. Wainwright, S. Marmorstein, and C. T. Morrison. Worldtree: A corpus of explanation graphs for elementary science questions supporting multi-hop inference. In LREC, 2018. Also arXiv:1802.03052.
  21. X. Zhou, M. Sap, S. Swayamdipta, N. A. Smith, and Y. Choi. Challenges in automated debiasing for toxic language detection. In EACL, 2021.
  22. A. Kalyan, A. Kumar, A. Chandrasekaran, A. Sabharwal, and P. Clark. How much coffee was consumed during find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tafjord-2021-general-purpose-question-answering/