Elamur Research
☰
home / papers / 2019 / Synthetic QA Corpora Generation with Roundtrip Consistency

Synthetic QA Corpora Generation with Roundtrip Consistency

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P19-1620.pdf

Summary

This paper asks whether QA systems can benefit from a pretraining task closer to question answering than masked language modeling. The authors build millions of synthetic question–answer–context triples from unlabeled Wikipedia text using three BERT-based models trained on labeled data. The first picks a candidate answer span from a passage. The second writes a question about that span. The third answers the generated question. A triple is kept only when the predicted answer agrees with the original span, a filter they call roundtrip consistency, inspired by back-translation in MT. A QA model is first pretrained on this synthetic corpus and then fine-tuned on SQuAD2 or NQ.

There are two variants. One only fine-tunes public BERT, so it is fully reproducible. The other uses a fully pretrained sequence-to-sequence Transformer for question generation. Both improve results. NQ reaches a new state of the art, and the strongest SQuAD2 ensemble nearly matches human scores. The work shows that cheap, filtered synthetic data can serve as an effective QA-specific pretraining signal.

Key points

  • Data sources: 8M unlabeled 512-token windows from the NQ training text (about 1B words). After roundtrip filtering, these yielded 2.4M SQuAD2-style and 3.2M NQ-style answerable examples. Added unanswerable examples brought the totals to 3M and 4M.
  • Fine-tuning-only setting on SQuAD2: BERT-Large rose from 78.7/81.9 to 81.2/84.0 dev EM/F1 when pretrained on combined SQuAD2+NQ synthetic data.
  • Full pretraining setting: a 50M-example synthetic corpus plus a 6-model ensemble scored 86.7 EM / 89.1 F1 on the SQuAD2 test set. Human performance is 86.8 / 89.5, and the system ranked third on the leaderboard (May 2019).
  • NQ results: over the BERT joint baseline, short-answer test F1 rose from 52.1 to 53.9 and long-answer test F1 from 66.2 to 66.8.
  • Value of the filter: in a manual check, 39% of kept triples were correct versus 16% of discarded ones. Filtered data consistently beat unfiltered data in the learning curves.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

146 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 143 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1620

@inproceedings{alberti2019synthetic,
  title = {{Synthetic QA Corpora Generation with Roundtrip Consistency}},
  author = {Alberti, Chris and Andor, Daniel and Pitler, Emily and Devlin, Jacob and Collins, Michael},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1620},
  url = {https://doi.org/10.18653/v1/p19-1620}
}

References

12 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chris Alberti, Kenton Lee, and Michael Collins. 2019. A bert baseline for the natural questions. arXiv Xinya Du and Claire Cardie. 2018. Harvesting paragraph-level question-answer pairs from wikipedia. arXiv preprint arXiv:1805.05942. find
  2. Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to ask: Neural question generation for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1342– 1352. Association for Computational Linguistics. find our card
  3. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489–500. Association for Computational Linguistics. find
  4. Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. 2016. Dual learning for machine translation. In Advances in Neural Information Processing Systems, pages 820–828. find
  5. Michael Heilman and Noah A Smith. 2010. Good question! statistical ranking for question generation. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 609–617. Association for Computational Linguistics. find
  6. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Rhinehart, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural… find our card
  7. Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, et al. 2018. Phrase-based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5039–5049. find
  8. Mike Lewis and Angela Fan. 2019. Generative question answering: Learning to answer the whole question. International Conference on Learning Representations (ICLR). find
  9. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you dont know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 784–789. find our card
  10. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 86–96. find
  11. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. Ernie: Enhanced representation through knowledge integration. CoRR, abs/1904.09223. find our card
  12. Zhilin Yang, Junjie Hu, Ruslan Salakhutdinov, and William Cohen. 2017. Semi-supervised qa with generative domain-adaptive nets. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1040–1050. find