Elamur Research⌕
☰
home / papers / 2022 / Domain Adaptation with Pre-trained Transformers for Query-Focused Abstractive Text Summarization

Domain Adaptation with Pre-trained Transformers for Query-Focused Abstractive Text Summarization

Paper · 2022 · Md Tahmid Rahman Laskar, Enamul Hoque and Jimmy Xiangji Huang

Summary

Query-focused abstractive summarization (QFAS) means generating a summary of one or more documents that answers a specific query. Its main obstacle is that labeled datasets for it are tiny. Multi-document inputs add a second problem: they are too long for transformers, whose self-attention cost grows quadratically with input length. The authors address both problems with domain adaptation from pre-trained transformers. A BERTSUM-style model is first trained on large generic summarization corpora and then fine-tuned on QFAS data with query relevance added. For document sets, they propose two strategies:

  • Weak supervision. They create weak per-document reference summaries and fine-tune on each document in turn.
  • Sentence filtering. A RoBERTa model selects the sentences most relevant to the query before abstractive summarization. A sequential fine-tuning scheme then lets every gold summary contribute.

Across six benchmarks (three single-document and three multi-document), the approach reaches new state-of-the-art scores on several datasets under both automatic and human evaluation. It also stays light enough to deploy in production.

Key points

  • The evaluation covers six datasets in total. The single-document group includes Debatepedia, and the multi-document group consists of DUC 2005, DUC 2006 and DUC 2007.
  • The analysis surfaced several findings:
    • Most Debatepedia queries turn out to be unrelated to their documents.
    • The type of encoder attention affects QFAS results.
    • The adaptation also helps abstractive answer generation on MS-MARCO.
  • The final summary-selection step uses RoBERTa Large fine-tuned for answer selection on MS-MARCO (153,725 queries). This setup beat variants tuned on TREC-QA, WikiQA and SemEvalCQA, reaching up to 44.7 R-1 on DUC 2007.
  • The full system is modest in size: the BERTSUM model takes 2.32GB and the RoBERTa model 1.43GB. Together they fit on a machine with no more than 4GB of RAM. Stronger models such as BART, PEGASUS and T5 can be swapped in when resources allow.
  • The code is publicly released at github.com/tahmedge/PreQFAS.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓York UniversityToronto · Canada15 works · 3,325 citations🎓York UniversityYork · United States4 works · 381 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Summarization · 243 works

Cite

Cite as Md Tahmid Rahman Laskar, Enamul Hoque and Jimmy Xiangji Huang. 2022. Domain Adaptation with Pre-trained Transformers for Query-Focused Abstractive Text Summarization. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00434

@article{laskar2022domain,
  title = {{Domain Adaptation with Pre-trained Transformers for Query-Focused Abstractive Text Summarization}},
  author = {Md Tahmid Rahman Laskar, Enamul Hoque and Jimmy Xiangji Huang},
  journal = {Computational Linguistics},
  year = {2022},
  doi = {10.1162/coli\_a\_00434},
  url = {https://doi.org/10.1162/coli_a_00434}
}

References

45 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdullah, Deen Mohammad and Yllias Chali. 2020. Towards generating query to perform query focused abstractive summarization using pre-trained model. In Proceedings of the 13th International Conference on Natural Language Generation, pages 80–85. find
  2. Antol, Stanislaw, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question Answering. In International Conference on Computer Vision (ICCV). pages 2425–2433. https://doi.org/10.1109/ICCV.2015.279 Aryal, Chudamani and Yllias Chali… ↗
  3. Beltagy, Iz, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150.
  4. Brown, Tom B., Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
  5. Chopra, Sumit, Michael Auli, and Alexander M. Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 93–98. https…
  6. Clark, Kevin, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555. find
  7. Deng, Yang, Wai Lam, Yuexiang Xie, Daoyuan Chen, Yaliang Li, Min Yang, and Ying Shen. 2019. Joint learning of answer selection and answer summary generation in community question answering. arXiv preprint arXiv:1911.09801. https://doi.org/10.1609/aaai.v34i05.6266 Devlin, Jacob, Ming-Wei Chang, Kent… ↗
  8. Dong, Li, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems, pages 13063–13075.
  9. Fabbri, Alexander Richard, Simeng Han, Haoyuan Li, Haoran Li, Marjan Ghazvininejad, Shafiq Joty, Dragomir Radev, and Yashar Mehdad. 2021. Improving zero and few-shot abstractive summarization with intermediate fine-tuning and data augmentation. In Proceedings of the 2021 Conference of the North Amer… find
  10. Huang, Xiangji and Qinmin Hu. 2009. A Bayesian learning approach to promoting diversity in ranking for biomedical information retrieval. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 307–314. https://doi.org/10.1145/1571941… ↗
  11. Ishigaki, Tatsuya, Hen-Hsen Huang, Hiroya Takamura, Hsin-Hsi Chen, and Manabu Okumura. 2020. Neural query-biased abstractive summarization using copying mechanism. In European Conference on Information Retrieval, pages 174–181. https://doi.org/10.1007/978-3 -030-45442-5 22 Kim, Dae Hyun, Enamul Hoqu… ↗
  12. Kulkarni, Sayali, Sheide Chammas, Wan Zhu, Fei Sha, and Eugene Ie. 2020. Aquamuse: Automatically generating datasets for query-based multi-document summarization. arXiv preprint arXiv:2010.12694. https://doi.org/10 .1007/978-3-030-86331-9 6 Kwiatkowski, Tom, Jennimaria Palomaki, Olivia Redfield, Mic…
  13. Lai, Tuan, Quan Hung Tran, Trung Bui, and Daisuke Kihara. 2019. A gated self-attention memory network for answer selection. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5955–…
  14. Laskar, Md Tahmid Rahman, Enamul Hoque, and Jimmy Huang. 2020a. Query focused abstractive summarization via incorporating query relevance and transfer learning with transformer models. In Canadian Conference on Artificial Intelligence, pages 342–348. https://doi.org/10 .1007/978-3-030-47358-7 35 Las… find
  15. Lewis, Mike, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. https://doi.org/10…
  16. Lin, Chin-Yew. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81.
  17. Liu, Xiaodong, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019a. Improving multi-task deep neural networks via knowledge distillation for natural language understanding. arXiv preprint arXiv:1904.09482. find
  18. Liu, Yang, Xiangji Huang, Aijun An, and Xiaohui Yu. 2007. ARSA: A sentiment-aware model for predicting sales performance using blogs. In Proceedings of the 30th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 607–614. https://doi.org/10 .1145/1277741.12…
  19. Louis, Annie and Ani Nenkova. 2013. Automatically assessing machine summary content without a gold standard. Computational Linguistics, 39(2):267–300. https://doi.org/10.1162/COLI_a_00123 ↗
  20. Ma, Shulei, Zhi-Hong Deng, and Yunlun Yang. 2016. An unsupervised multi-document summarization framework based on neural document model. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1514–1523. Nallapati, Ramesh, Bowen Zhou, Ci…
  21. Nema, Preksha, Mitesh M Khapra, Anirban Laha, and Balaraman Ravindran. 2017. Diversity driven attention model for query-based abstractive summarization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1063–1072. https://doi.or… ↗
  22. Nishida, Kyosuke, Itsumi Saito, Kosuke Nishida, Kazutoshi Shinoda, Atsushi Otsuka, Hisako Asano, and Junji Tomita. 2019. Multi-style generative reading comprehension. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2273–2284. https://doi.org/10.18653… find
  23. Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318. https://doi.org/10.3115/1073083 .1073135 Pasunuru, Ramakanth,… ↗
  24. Paulus, Romain, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations.
  25. Peters, Matthew E., Sebastian Ruder, and Noah A. Smith. 2019. To tune or not to tune? Adapting pretrained representations to diverse tasks. In Proceedings of the 4th Workshop on Representation Learning for NLP, pages 7–14. https://doi.org/10 .18653/v1/W19-4302 Qi, Weizhen, Yu Yan, Yeyun Gong, Dayihe…
  26. Raffel, Colin, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
  27. Ramponi, Alan and Barbara Plank. 2020. Neural unsupervised domain adaptation in NLP—a survey. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6838–6855. https://doi.org/10.18653/v1/2020 .coling-main.603 Reiter, Ehud. 2018. A structured review of the validity o… ↗
  28. Sanh, Victor, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2021. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207.
  29. Savery, Max, Asma Ben Abacha, Soumya Gayen, and Dina Demner-Fushman. 2020. Question-driven summarization of answers to consumer health questions. Scientific Data, 7(1):1–9. https://doi.org/10.1038/s41597-020-00667-z See, Abigail, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Sum… ↗
  30. Su, Dan, Yan Xu, Tiezheng Yu, Farhad Bin Siddique, Elham J. Barezi, and Pascale Fung. 2020. CAiRE-COVID: A question answering and query-focused multi-document summarization system for COVID-19 scholarly information management. arXiv preprint arXiv:2005.03975. https://doi.org/10.18653/v1/2020 .nlpco… ↗
  31. Tay, Yi, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2020. Efficient transformers: A survey. arXiv preprint arXiv:2009.06732.
  32. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008.
  33. Wan, Xiaojun and Jianguo Xiao. 2009. Graph-based multi-modality learning for topic-focused multi-document summarization. In Twenty-First International Joint Conference on Artificial Intelligence. find
  34. Wan, Xiaojun and Jianmin Zhang. 2014. CTSUM: extracting more certain summaries for news articles. In Proceedings of the 37th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 787–796. https://doi.org/10.1145/2600428.2609559 Wang, Dingding, Shenghuo Zhu,… — [doi:10.1145/2600428.2609559](https://doi.org/10.1145/2600428.2609559) ↗
  35. Wang, Sinong, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768. find
  36. Wang, Yizhong, Kai Liu, Jing Liu, Wei He, Yajuan Lyu, Hua Wu, Sujian Li, and Haifeng Wang. 2018. Multi-passage machine reading comprehension with cross-passage answer verification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pag…
  37. Xie, Yujia, Tianyi Zhou, Yi Mao, and Weizhu Chen. 2020. Conditional self-attention for query-based summarization. arXiv preprint arXiv:2002.07338. — [arXiv:2002.07338](https://arxiv.org/abs/2002.07338) find
  38. Xu, Yumo and Mirella Lapata. 2020a. Abstractive query focused summarization with query-free resources. arXiv preprint arXiv:2012.14774. https://doi.org/10 .18653/v1/2021.acl-long.475 Xu, Yumo and Mirella Lapata. 2020b. Coarse-to-fine query focused multi-document summarization. In Proceedings of the… find
  39. Yang, Yi, Wen-tau Yih, and Christopher Meek. 2015. WikiQA: A challenge dataset for open-domain question answering. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2013–2018. https://doi.org/10.18653/v1/D15-1237 Yao, Jin ge, Xiaojun Wan, and Jianguo X… ↗
  40. Yao, Jin Ge, Xiaojun Wan, and Jianguo Xiao. 2017. Recent advances in document summarization. Knowledge and Information Systems, 53(2):297–336. https://doi.org/10.1007/s10115-017-1042-4 Young, Tom, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria. 2017. Recent trends in deep learning based natur… ↗
  41. Yu, Xiaohui, Yang Liu, Xiangji Huang, and Aijun An. 2012. Mining online reviews for predicting sales performance: A case study in the movie domain. IEEE Transactions on Knowledge and Data Engineering, 24(4):720–734. https://doi.org/10 .1109/TKDE.2010.269 Yuan, Weizhe, Graham Neubig, and Pengfei Liu.…
  42. Zaheer, Manzil, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big Bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062.
  43. Zhang, Jingqing, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2019a. PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization. arXiv preprint arXiv:1912.08777.
  44. Zhang, Tianyi, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019b. BERTscore: Evaluating text generation with BERT. In International Conference on Learning Representations. find
  45. Zhong, Ming, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan, et al. 2021. QMSum: A new benchmark for query-based multi-domain meeting summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lang…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/laskar-2022-domain-adaptation-pre-trained-transformers/