Elamur Research⌕
☰
home / papers / 2021 / Controllable Summarization with Constrained Markov Decision Process

Controllable Summarization with Constrained Markov Decision Process

Paper · 2021 · Hou Pong Chan, Lu Wang and Irwin King

Summary

Token-based controllable summarizers such as ControlSum let a user request a property of the output, for example a word budget. Their maximum-likelihood training, however, gives no direct signal against breaking that request, so many outputs still miss it. Reinforcement learning with a plain MDP could add penalty terms, but balancing several weighted rewards demands heavy tuning. The authors instead recast training as a Constrained Markov Decision Process (CMDP). A summarization metric acts as the reward that pulls outputs toward the references, while separate constraints forbid violating the user's requirement.

They design constraints for three attributes:

  • Length, using disjoint length bins.
  • Covered entities, using a QA-based check that the summary conveys the key facts about the requested entities.
  • Abstractiveness, using three levels based on extractive fragment density.

The framework is used to fine-tune a pointer-generator RNN and DistilGPT2 on benchmarks including CNN/DM and Newsroom-b. Both models follow the requirements more reliably, and they gain ROUGE when given the reference control tokens. Human judges confirm the summaries are informative and compliant. The main benefit is that fragile reward weighting is replaced by a few interpretable thresholds.

Key points

  • Tuning burden: CMDP needs only one threshold, a QA-F1 value close to 1 for entity control, and none for length or abstractiveness. The length-bin constraint is simply set to 0. An equivalent MDP setup would need 2, 3 and 3 penalty weights for length, entity and abstractiveness control respectively.
  • Base models: the pointer-generator network (See et al., 2017) and DistilGPT2 (Sanh et al., 2019). DistilGPT2 was chosen because it is smaller than GPT2.
  • Abstractiveness ranking: in human evaluation, partial-match scores were very high. Exact-match scores were lower on the more extractive CNN/DM than on Newsroom-b, but still exceeded 60%.
  • Summary quality on Newsroom-b: at the lowest abstractiveness bin, fluency was about 4.8/5 and faithfulness 96–98%. At higher bins, relevance and faithfulness dropped sharply, with faithfulness around 52–66%, since paraphrasing is harder than copying. Krippendorff's α agreement was 0.51, 0.37 and 0.40 for fluency, relevance and faithfulness.
  • Code: released at github.com/kenchan0226/control-sum-cmdp. The paper itself is published under CC-BY 4.0 in TACL vol. 9 (2021).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of MichiganAnn Arbor · United States70 works · 13,693 citations🎓Chinese University of Hong KongHong Kong · China126 works · 14,073 citations🎓University of MacauMacao · China33 works · 2,223 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Summarization · 232 works

Cite

Cite as Hou Pong Chan, Lu Wang and Irwin King. 2021. Controllable Summarization with Constrained Markov Decision Process. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00423

@article{chan2021controllable,
  title = {{Controllable Summarization with Constrained Markov Decision Process}},
  author = {Hou Pong Chan, Lu Wang and Irwin King},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00423},
  url = {https://doi.org/10.1162/tacl_a_00423}
}

References

70 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eitan Altman. 1999. Constrained Markov Decision Processes, volume 7. CRC Press. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7–9, 2015, Conference Track Proceedings.
  3. Richard Bellman. 1957. A Markovian decision process. Journal of Mathematics and Mechanics, pages 679–684. find
  4. Dimitri P. Bertsekas. 1997. Nonlinear programming. Journal of the Operational Research Society, 48(3):334–334. https://doi.org/10.1057/palgrave.jors.2600425 — [doi:10.1057/palgrave.jors.2600425](https://doi.org/10.1057/palgrave.jors.2600425) ↗
  5. Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. 2020. With little power comes great responsibility. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16–20, 2020, pages 9263–9274. Asso… find
  6. Asli Çelikyilmaz, Antoine Bosselut, Xiaodong He, and Yejin Choi. 2018. Deep communicating agents for abstractive summarization. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orle…
  7. Hou Pong Chan, Wang Chen, and Irwin King. 2020. A unified dual-view model for review summarization and sentiment classification with inconsistency loss. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, Chin… find
  8. Hou Pong Chan and Irwin King. 2021. A condense-then-select strategy for text summarization. Knowledge-Based Systems, page 107235. https://doi.org/10.1016/j.knosys .2021.107235 — [doi:10.1016/j.knosys](https://doi.org/10.1016/j.knosys) ↗
  9. Xiuying Chen, Shen Gao, Chongyang Tao, Yan Song, Dongyan Zhao, and Rui Yan. 2018. Iterative document representation learning towards summarization with polishing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 – November 4, 20… find
  10. Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15–20, 2018, Volume 1: Long Papers, pages 675–686. Associa…
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  12. Angela Ellis. 2002. Document Understanding Conferences (DUC) 2002 Data. find
  13. Matan Eyal, Tal Baumel, and Michael Elhadad. 2019. Question answering as an automatic evaluation metric for news article summarization. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, M…
  14. Angela Fan, David Grangier, and Michael Auli. 2018. Controllable abstractive summarization. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, NMT@ACL 2018, Melbourne, Australia, July 20, 2018, pages 45–54.
  15. Jessica Ficler and Yoav Goldberg. 2017. Controlling linguistic style aspects in neural language generation. CoRR, abs/1707.02633.
  16. Sebastian Gehrmann, Yuntian Deng, and Alexander M. Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 4098–4109. Association for Computational Linguistics.
  17. Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orlean…
  18. Jiatao Gu, Qi Liu, and Kyunghyun Cho. 2019. Insertion-based decoding with automatically inferred generation order. Transactions of the Association for Computational Linguistics, 7:661–676.
  19. Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December…
  20. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. https://doi.org/10 .1162/neco.1997.9.8.1735, PubMed: 9377276 find
  21. Kai Hong, John M. Conroy, Benoı̂t Favre, Alex Kulesza, Hui Lin, and Ani Nenkova. 2014. A repository of state of the art and competitive baseline summaries for generic news summarization. In Proceedings of the Ninth International Conference on Language Resources and Evaluation, LREC 2014, Reykjavik,… find
  22. Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020. spaCy: Industrial-strength Natural Language Processing in Python. find
  23. Luyang Huang, Lingfei Wu, and Lu Wang. 2020. Knowledge graph-augmented abstractive summarization with semantic-driven cloze reward. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5–10, 2020, pages 5094–5107. https://doi .org/10.1865… find
  24. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7–9, 2015, Conference Track Proceedings.
  25. Catherine Kobus, Josep Crego, and Jean Senellart. 2017. Domain control for neural machine translation. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, pages 372–378, Varna, Bulgaria. INCOMA Ltd. find
  26. Wojciech Kryscinski, Romain Paulus, Caiming Xiong, and Richard Socher. 2018. Improving abstraction in text summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 1808–1817. https://doi.org /10.… ↗
  27. Philippe Laban, Andrew Hsi, John Canny, and Marti A. Hearst. 2020. The summary loop: Learning to write abstractive summaries without examples. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5–10, 2020, pages 5135–5150. https://doi .… find
  28. Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019. CTRL: A conditional transformer language model for controllable generation. CoRR, abs/1909.05858.
  29. Logan Lebanoff, Kaiqiang Song, Franck Dernoncourt, Doo Soon Kim, Seokhwan Kim, Walter Chang, and Fei Liu. 2019. Scoring sentence singletons and pairs for abstractive summarization. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July…
  30. Yuta Kikuchi, Graham Neubig, Ryohei Sasano, Hiroya Takamura, and Manabu Okumura. 2016. Controlling output length in neural Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. Text Summarization Branches Out.
  31. Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3–7, 2019…
  32. Preksha Nema, Mitesh M. Khapra, Anirban Laha, and Balaraman Ravindran. 2017. Diversity driven attention model for query-based abstractive summarization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 – August 4, Volume… find
  33. Yizhu Liu, Zhiyi Luo, and Kenny Q. Zhu. 2018. Controlling length in abstractive summarization using a convolutional neural network. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 4110–4119. find
  34. Tong Niu and Mohit Bansal. 2018. Polite dialogue generation without parallel data. TACL, 6:373–389. https://doi.org/10.1162/tacl_a_00027 ↗
  35. Ilya Loshchilov and Frank Hutter. 2017. Fixing weight decay regularization in Adam. CoRR, abs/1711.05101. find
  36. Takuya Makino, Tomoya Iwakura, Hiroya Takamura, and Manabu Okumura. 2019. Global optimization under length constraint for neural text summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1039–1048, Florence, Italy. Association for Computati… find
  37. Ning Miao, Hao Zhou, Lili Mou, Rui Yan, and Lei Li. 2019. CGMH: constrained sentence generation by metropolis-hastings sampling. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The N… find
  38. Ramesh Nallapati, Bowen Zhou, Cı́cero Nogueira dos Santos, Çaglar Gülçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016, Berlin, Germany, August…
  39. Ramakanth Pasunuru and Mohit Bansal. 2018. Multi-reward reinforced summarization with saliency and entailment. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, Ju…
  40. Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings.
  41. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). find
  42. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers, pages 784–789. https:/…
  43. Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21–26, 2017, pages 1179–1195. https://doi.org/10 .11…
  44. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. CoRR, abs/1910.01108. find
  45. Raphael Schumann, Lili Mou, Yao Lu, Olga Vechtomova, and Katja Markert. 2020. Discrete optimization for unsupervised sentence summarization with word-level extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5–10, 2020, pages… find
  46. Thomas Scialom, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2019. Answers unite! unsupervised metrics for reinforced summarization models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural La…
  47. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 1073… find
  48. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Controlling politeness in neural machine translation via side constraints. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California,…
  49. Eva Sharma, Luyang Huang, Zhe Hu, and Lu Wang. 2019. An entity-driven framework for abstractive summarization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong… find
  50. Kaiqiang Song, Bingqing Wang, Zhe Feng, Ren Liu, and Fei Liu. 2020. Controlling the amount of verbatim copying in abstractive summarization. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8902–8909. AAAI Press. https://doi.o… — [doi:10.1609/aaai.v34i05.6420](https://doi.org/10.1609/aaai.v34i05.6420) ↗
  51. Kaiqiang Song, Lin Zhao, and Fei Liu. 2018. Structure-infused copy mechanisms for abstractive summarization. In Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20–26, 2018, pages 1717–1729. find
  52. Qing Sun, Stefan Lee, and Dhruv Batra. 2017. Bidirectional beam search: Forward-backward inference in neural sequence models for fill-in-the-blank image captioning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July21–26, 2017, pages 7215–7223. htt… — [doi:10.1109/cvpr.2017.763](https://doi.org/10.1109/cvpr.2017.763) find
  53. Richard S. Sutton and Andrew G. Barto. 1998. Reinforcement Learning - an Introduction. Adaptive computation and machine learning. MIT Press. find
  54. Shunsuke Takeno, Masaaki Nagata, and Kazuhide Yamamoto. 2017. Controlling target features in neural machine translation via prefix constraints. In Proceedings of the 4th Workshop on Asian Translation, WAT@IJCNLP 2017, Taipei, Taiwan, November 27–December 1, 2017, pages 55–63. find
  55. Chen Tessler, Daniel J. Mankowitz, and Shie Mannor. 2019. Reward constrained policy optimization. In International Conference on Learning Representations (ICLR). find
  56. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4–9 December 2017,…
  57. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910. 03771. https://doi.org/10.18653/v1/2020.emnlp-demos.6 ↗
  58. Xiaojun Wan. 2008. Using only cross-document relationships for both generic and topic-focused multi-document summarizations. Information Retrieval, 11(1):25–49. https://doi.org/10.1007/s10791-007-9037-5 — [doi:10.1007/s10791-007-9037-5](https://doi.org/10.1007/s10791-007-9037-5) ↗
  59. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020a. PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13–18 July 2020, Virtual Event, volume 119 of Proceedings of Machin…
  60. Xiaojun Wan. 2009. Topic analysis for topic-focused multi-document summarization. In Proceedings of the 18th ACM Conference on Information and Knowledge Management, CIKM 2009, Hong Kong, China, November 2–6, 2009, pages 1609–1612. ACM. find
  61. Xiaojun Wan and Jianguo Xiao. 2009. Graph-based multi-modality learning for topic-focused multi-document summarization. In IJCAI 2009, Proceedings of the 21st International Joint Conference on Artificial Intelligence, Pasadena, California, USA, July 11–17, 2009, pages 1586–1591. find
  62. Xiaojun Wan, Jianwu Yang, and Jianguo Xiao. 2007. Manifold-ranking based topic-focused multi-document summarization. In IJCAI 2007, Proceedings of the 20th International Joint Conference on Artificial Intelligence, Hyderabad, India, January 6–12, 2007, pages 2903–2908. find
  63. Xiaojun Wan and Jianmin Zhang. 2014. CT-SUM: extracting more certain summaries for news articles. In The 37th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’14, Gold Coast, QLD, Australia - July 06 - 11, 2014, pages 787–796. ACM. find
  64. Lu Wang, Hema Raghavan, Vittorio Castelli, Radu Florian, and Claire Cardie. 2013. A sentence compression based framework to query-focused multi-document summarization. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, ACL 2013, 4–9 August 2013, Sofia, Bulgar… find
  65. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. find
  66. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020b. BERTScore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26–30, 2020. OpenReview.net. find
  67. Yuhao Zhang, Derek Merck, Emily Bao Tsai, Christopher D. Manning, and Curtis Langlotz. 2020c. Optimizing the factual correctness of a summary: A study of summarizing radiology reports. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July…
  68. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International…
  69. Wanrong Zhu, Zhiting Hu, and Eric P. Xing. 2019. Text infilling. CoRR, abs/1901.00158. — [arXiv:1901.00158](https://arxiv.org/abs/1901.00158) find
  70. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2020. Fine-tuning language models from human preferences. CoRR, abs/1909.08593. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chan-2021-controllable-summarization-constrained-markov-decision/