Elamur Research
☰
home / papers / 2020 / WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarization

WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarization

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.findings-emnlp.360.pdf

Summary

Abstractive summarization research has stayed mostly monolingual because existing multilingual corpora (MultiLing'13/'15, Global Voices) are very small or lack parallel article–summary pairs. Collecting such data through crowd-sourcing is also expensive and subjective. The authors build WikiLingua from WikiHow how-to guides in 18 languages. For each method in a guide, the detailed step paragraphs form the article, and the one-sentence step summaries form the reference summary. WikiHow reuses the same illustration for a given step in every language version, so the images serve as anchors for aligning steps across languages, which yields gold-standard cross-lingual pairs. Using mBART, the authors benchmark standard pipelines that translate either before or after summarizing. The task is to produce English summaries from Spanish, Russian, Turkish and Vietnamese articles. They also introduce a direct cross-lingual model whose pre-training step draws on machine-translated and artificially generated examples. This direct model clearly beats the baselines and needs no translation at inference time, so it is cheaper to run. The resource makes cross-lingual and multilingual summarization research possible at a much larger scale.

Key points

  • WikiLingua covers 18 languages and contains 141,457 English article–summary pairs. The other 17 languages average 42,783 articles aligned to English, ranging from 113,215 for Spanish to 4,503 for Turkish.
  • Even the smallest language, Turkish, is about ten times larger than any language in earlier cross-lingual datasets. The per-language article count is roughly two orders of magnitude larger than Global Voices.
  • Each guide is written and edited by about 23 people and reviewed by about 16 on average. Non-English versions are mostly human translations checked by WikiHow's translation team.
  • The experiments fine-tune mBART, which is pre-trained on 25 languages, for four source languages: es, ru, tr and vi. The Spanish split, for example, has 81,514 train, 9,057 validation and 22,643 test examples. English articles sharing topics with any non-English test article are kept in the test set.
  • The multi-task round-trip-translation approach of Zhu et al. performed worse than every baseline except lead on this data, with ROUGE-L F1 between about 19 and 23. The dataset and splits are released at github.com/esdurmus/Wikilingua.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

123 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 115 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.findings-emnlp.360 · arXiv 2010.03093

@inproceedings{ladhak2020wikilingua,
  title = {{WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarization}},
  author = {Ladhak, Faisal and Durmus, Esin and Cardie, Claire and McKeown, Kathleen},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.360},
  eprint = {2010.03093},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.findings-emnlp.360}
}

References

41 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sebastian Gehrmann, Yuntian Deng, and Alexander M Rush. 2018. Bottom-up abstractive summarization. arXiv preprint arXiv:1808.10792. find our card
  2. Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604. find
  3. Ronald Brandow, Karl Mitze, and Lisa F. Rau. 1995. Automatic condensation of electronic publications Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pa… find
  4. Michael Elhadad, Sabino Miranda-Jiménez, Josef Steinberger, and George Giannakopoulos. 2013. Multi-document multilingual summarization corpus preparation, part 2: Czech, Hebrew and Spanish. In Proceedings of the MultiLing 2013 Workshop on Multilingual Multi-document Summarization, pages 13–19, Sofi… find our card
  5. George Giannakopoulos. 2013. Multi-document multilingual summarization and evaluation tracks in ACL 2013 MultiLing workshop. In Proceedings of the MultiLing 2013 Workshop on Multilingual Multi-document Summarization, pages 20–28, Sofia, Bulgaria. Association for Computational Linguistics. find
  6. George Giannakopoulos, Jeff Kubina, John Conroy, Josef Steinberger, Benoit Favre, Mijail Kabadjov, Udo Kruschwitz, and Massimo Poesio. 2015. find
  7. MultiLing 2015: Multilingual summarization of single and multi-documents, on-line fora, and call-center conversations. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 270–274, Prague, Czech Republic. Association for Computational Linguistics. find
  8. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS’15, page 1693–1701,… find our card
  9. Vu Cong Duy Hoang, Philipp Koehn, Gholamreza Haffari, and Trevor Cohn. 2018. Iterative back-translation for neural machine translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 18–24, Melbourne, Australia. Association for Computational Linguistics. find our card
  10. Baotian Hu, Qingcai Chen, and Fangze Zhu. 2015. LC-STS: A large scale Chinese short text summarization dataset. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1967–1972, Lisbon, Portugal. Association for Computational Linguistics. find
  11. Mahnaz Koupaee and William Yang Wang. 2018. Wikihow: A large scale text summarization dataset. CoRR, abs/1810.09305. — [arXiv:1810.09305](https://arxiv.org/abs/1810.09305) find
  12. Jeff Kubina, John Conroy, and Judith Schlesinger. 2013. find
  13. ACL 2013 MultiLing pilot overview. In Proceedings of the MultiLing 2013 Workshop on Multilingual Multi-document Summarization, pages 29–38, Sofia, Bulgaria. Association for Computational Linguistics. find
  14. Anton Leuski, Chin-Yew Lin, Liang Zhou, Ulrich Germann, Franz Josef Och, and Eduard Hovy. 2003. Cross-lingual c*st*rd: English access to hindi information. ACM Transactions on Asian Language Information Processing, 2(3):245–269. find
  15. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019a. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. ArXiv, abs/1910.13461. find our card
  16. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019b. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. find our card
  17. Lei Li, Corina Forascu, Mahmoud El-Haj, and George Giannakopoulos. 2013. Multi-document multilingual summarization corpus preparation, part 1: Arabic, English, Greek, Chinese, Romanian. In Proceedings of the MultiLing 2013 Workshop on Multilingual Multi-document Summarization, pages 1–12, Sofia, Bul… find
  18. Jung-Min Lim, In-Su Kang, and Jong-Hyeok Lee. 2004. Multi-document summarization using cross-language texts. In NTCIR. find
  19. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. find
  20. Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730–3740, Hong Kong, China. Assoc… find
  21. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. find
  22. Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gu̇lçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280–290, Berlin, Germany. Association… find
  23. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! Topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium. find
  24. Khanh Nguyen and Hal Daumé III. 2019. Global voices: Crossing borders in automatic news summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 90–97, Hong Kong, China. Association for Computational Linguistics. find
  25. Constantin Orăsan and Oana Andreea Chiorean. 2008. Evaluation of a cross-lingual Romanian-English multi-document summariser. In LREC 2008. find
  26. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations. find our card
  27. Jessica Ouyang, Boya Song, and Kathy McKeown. 2019. A robust abstractive system for cross-lingual summarization. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages… find
  28. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379–389, Lisbon, Portugal. Association for Computational Linguistics. find
  29. Sebastian Schuster, Sonal Gupta, Rushin Shah, and Mike Lewis. 2019. Cross-lingual transfer learning for multilingual task oriented dialog. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long… find our card
  30. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073– 1083, Vancouver, Canada. Association for Compu… find
  31. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86–96, Berlin, Germany. Association for Computational L… find
  32. Shi-qi Shen, Yun Chen, Cheng Yang, Zhi-yuan Liu, and Mao-song Sun. 2018. Zero-shot cross-lingual neural headline generation. IEEE/ACM Trans. Audio, Speech and Lang. Proc., 26(12):2319–2327. for Computational Linguistics, pages 917–926, Uppsala, Sweden. Association for Computational Linguistics. find
  33. Xiaojun Wan, Fuli Luo, Xue Sun, Songfang Huang, and Jin-ge Yao. 2019. Cross-language document summarization via extraction and ranking of multiple summaries. Knowledge and Information Systems, 58(2):481–499. find
  34. Jin-ge Yao, Xiaojun Wan, and Jianguo Xiao. 2015. Phrase-based compressive cross-language summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 118–127, Lisbon, Portugal. Association for Computational Linguistics. find
  35. J. Zhang, Y. Zhou, and C. Zong. 2016. Abstractive cross-language summarization via translation model enhanced predicate argument structure fusing. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 24(10):1842–1853. find
  36. Junnan Zhu, Haoran Li, Tianshang Liu, Yu Zhou, Jiajun Zhang, and Chengqing Zong. 2018. MSMO: Multimodal summarization with multimodal output. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4154–4164, Brussels, Belgium. Association for Computational L… find
  37. Junnan Zhu, Qian Wang, Yining Wang, Yu Zhou, Jiajun Zhang, Shaonan Wang, and Chengqing Zong. 2019. NCLS: Neural cross-lingual summarization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proces… find
  38. Kaiqiang Song, Bingqing Wang, Zhe Feng, Liu Ren, and Fei Liu. 2020. Controlling the amount of verbatim copying in abstractive summarization. In Proceedings of the AAAI Conference on Artificial Intelligence. find
  39. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2692–2700. Curran Associates, Inc. find
  40. Xiaojun Wan. 2011. Bilingual co-training for sentiment classification of Chinese product reviews. Computational Linguistics, 37(3):587–616. find
  41. Xiaojun Wan, Huiying Li, and Jianguo Xiao. 2010. Cross-language document summarization based on machine translation quality prediction. In Proceedings of the 48th Annual Meeting of the Association find