Elamur Research⌕
☰
home / papers / 2021 / Evaluating Document Coherence Modeling

Evaluating Document Coherence Modeling

Paper · 2021 · Aili Shen, Meladel Mistica, Bahar Salehi et al.

Summary

The paper asks whether pretrained language models genuinely understand document coherence or only exploit surface cues. The authors frame this as a sentence intrusion detection task. One sentence in a document is swapped for a similar sentence from another document. A model must then decide whether the document contains an intruder (coarse-grained) and, if so, locate it (fine-grained). Earlier coherence datasets were either too small or lacked the intruder's position, so the authors built INSteD, a large English corpus from Wikipedia and CNN news. They used adversarial filtering to reduce artefacts and collected human annotations for the Wikipedia portion. They benchmarked several pretrained LMs, compared them with human judgments, and wrote a hand-crafted linguistic probe set. The models score highly when trained and tested on the same domain but drop sharply across domains. They also miss certain coherence signals. The work therefore offers a resource and diagnostic tools for discourse-level evaluation, with possible uses in explainable essay scoring.

Key points

  • INSteD dataset: more than 170K documents, about 106K from Wikipedia and 72K from CNN. Documents vary in length and include both coherent and incoherent texts. Each intruder sentence's position is marked.
  • Models tested: BERT-Large, XLNet-Large, RoBERTa-Large and ALBERT-xxLarge, each evaluated in-domain and cross-domain (for example, trained on CNN and tested on Wikipedia).
  • Linguistic probes: eight types of hand-made edits:
    • Gender
    • Animacy↓ and Animacy↑
    • Demonstrative
    • Conjunction
    • Past to Future
    • Negation
    • Number
  • Probe results: models and Amazon Mechanical Turk annotators both gained a lot from probes such as Gender, Animacy and Negation. Both barely reacted to Demonstrative and Conjunction edits; for example, XLNet-Large and RoBERTa-Large showed a ΔF1 of 0.0 on Demonstrative.
  • Comparison with humans: in-domain, ALBERT-xxLarge and RoBERTa-Large match human performance. Cross-domain, they generally fall well below it.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of MelbourneMelbourne · Australia100 works · 22,618 citations🏢ByteDanceBeijing · China41 works · 4,180 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPhilosophy · 284 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Aili Shen, Meladel Mistica, Bahar Salehi, Hang Li, Timothy Baldwin and Jianzhong Qi. 2021. Evaluating Document Coherence Modeling. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00388

@article{shen2021evaluating,
  title = {{Evaluating Document Coherence Modeling}},
  author = {Aili Shen, Meladel Mistica, Bahar Salehi, Hang Li, Timothy Baldwin and Jianzhong Qi},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00388},
  url = {https://doi.org/10.1162/tacl_a_00388}
}

References

83 references, 35 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wafia Adouane, Jean-Philippe Bernardy, and Simon Dobnik. 2018a. Improving neural network performance by injecting background knowledge: Detecting code-switching and borrowing in Algerian texts. In Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching, pages 20–28… find
  2. Yonatan Belinkov and James Glass. 2019. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49–72. DOI: https://doi.org/10.1162/tacl a 00254 ↗
  3. Terra Blevins, Omer Levy, and Luke Zettlemoyer. 2018. Deep RNNs encode soft hierarchical syntax. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 14–19. DOI: https://doi.org/10.18653/v1/P18-2003 ↗
  4. Wafia Adouane, Simon Dobnik, Jean-Philippe Bernardy, and Nasredine Semmar. 2018b. A comparison of character neural language model and bootstrapping for language identification in multilingual noisy texts. In Proceedings of the Second Workshop on Subword/Character LEvel Models, pages 22–31. DOI: http… find
  5. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642. DOI: https://doi.org/10.18653/v1/D15 -1075 ↗
  6. Malihe Alikhani, Sreyasi Nag Chowdhury, Gerard de Melo, and Matthew Stone. 2019. CITE: A corpus of image-text discourse relations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Sho…
  7. Jamie Callan, Mark Hoy, Changkuk Yoo, and Le Zhao. 2009. Clueweb09 data set. https://lemurproject.org/clueweb09/. Accessed: 15.12.2019. ↗
  8. Nicholas M. Asher and Alex Lascarides. 2003. Logics of Conversation, Studies in Natural Language Processing. Cambridge University Press. find
  9. Khyathi Chandu, Eric Nyberg, and Alan W Black. 2019. Storyboarding of recipes: Grounded contextual generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6040–6046. DOI: https://doi.org/10.18653/v1/P19 -1606 ↗
  10. Regina Barzilay and Mirella Lapata. 2005. Modeling local coherence: An entity-based approach. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics, pages 141–148. DOI: https://doi.org/10.3115/1219840 .1219858 — [doi:10.3115/1219840](https://doi.org/10.3115/1219840) ↗
  11. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870–1879. DOI: https://doi.org/10.18653/v1/P17 -1171 ↗
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  13. Mingda Chen, Zewei Chu, and Kevin Gimpel. 2019. Evaluation benchmarks and learning criteria for discourse-aware sentence representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processin… find
  14. Micha Elsner, Joseph Austerweil, and Eugene Charniak. 2007. A unified local and global model for discourse coherence. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages 436–443.
  15. Micha Elsner and Eugene Charniak. 2011. Extending the entity grid with entity-specific features. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 125–129. find
  16. Minhao Cheng, Jinfeng Yi, Pin-Yu Chen, Huan Zhang, and Cho-Jui Hsieh. 2020. Seq2sick: Evaluating the robustness of sequence-to-sequence models with adversarial examples. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, pages 3601–3608. DOI: https://doi.org /10.1609/aaa… ↗
  17. Orphée De Clercq, Véronique Hoste, Bart Desmet, Philip van Oosten, Martine De Cock, and Lieve Macken. 2014. Using the crowd for readability prediction. Natural Language Engineering, 20(3):293–325. DOI: https://doi.org/10.1017/S1351324912000344 — [doi:10.1017/s1351324912000344](https://doi.org/10.1017/s1351324912000344) ↗
  18. Angela Fan, Mike Lewis, and Yann Dauphin. 2019. Strategies for structuring story generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2650–2660. find
  19. Herbert Paul Grice. 2002. Logic and conversation. Foundations of Cognitive Psychology, 719–732. find
  20. Camille Guinaudeau and Michael Strube. 2013. Graph-based local coherence modeling. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 93–103. find
  21. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680. DOI: https:…
  22. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume…
  23. Alexis Conneau, German Kruszewski, Guillaume Lample, Loı̈c Barrault, and Marco Baroni. 2018. What you can cram into a single $&!\#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long…
  24. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. find
  25. Teaching machines to read and comprehend. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, pages 1693–1701. auto-correlational neural networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 461… find
  26. John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2733–2743. DOI: https://doi.org /10.1865… ↗
  27. Yangfeng Ji and Noah A. Smith. 2017. Neural discourse structure for text categorization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 996–1005.
  28. Mark Johnson and Eugene Charniak. 2004. A TAG-based noisy-channel model of speech repairs. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, pages 33–39. DOI: https://doi.org/10.3115/1218955.1218960 ↗
  29. John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129…
  30. Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015. Skip-thought vectors. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, pages 3294–3302. find
  31. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. DOI: https://doi.org/10.1162/neco.1997.9.8.1735, PMID: 9377276 ↗
  32. Adhiguna Kuncoro, Chris Dyer, John Hale, Dani Yogatama, Stephen Clark, and Phil Blunsom. 2018. LSTMs can learn syntax-sensitive dependencies well, but modeling structure makes them better. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape…
  33. Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. 2018. Learning to write with cooperative discriminators. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1638–1649. DOI: https://doi.org/1…
  34. Alice Lai and Joel Tetreault. 2018. Discourse coherence in the wild: A dataset, evaluation and methods. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, pages 214–223. DOI: https://doi.org/10.18653/v1/W18-5023 ↗
  35. Jennifer Hu, Sherry Y. Chen, and Roger P. Levy. 2020a. A closer look at the performance of neural language models on reflexive anaphor licensing. Proceedings of the Society for Computation in Linguistics, 3(1):382–392.
  36. Junjie Hu, Yu Cheng, Zhe Gan, Jingjing Liu, Jianfeng Gao, and Graham Neubig. 2020b. What makes a good story? Designing composite rewards for visual storytelling. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, pages 7969–7976. DOI: https://doi.org/10.1609/aaai .v34i05… ↗
  37. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In Proceedings of the 8th International Conference on Learning Representations.
  38. Mirella Lapata. 2003. Probabilistic text structuring: Experiments with sentence ordering. In Proceedings of the 41st Annual Meeting Paria Jamshid Lou, Peter Anderson, and Mark Johnson. 2018. Disfluency detection using of the Association for Computational Linguistics, pages 545–552. DOI: https://doi…
  39. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202. DOI: https://doi.org/10.18653/v1/D18-1151 ↗
  40. Jey Han Lau, Alexander Clark, and Shalom Lappin. 2015. Unsupervised prediction of acceptability judgements. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pa… find
  41. Deepthi Mave, Suraj Maharjan, and Thamar Solorio. 2018. Language identification and analysis of code-switched social media text. In Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching, pages 51–61. DOI: https://doi.org/10.18653/v1/W18-3206 ↗
  42. Jiwei Li and Dan Jurafsky. 2017. Neural net models of open-domain discourse coherence. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 198–209.
  43. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448. DOI: https://doi.org/10.18653/v1/P19 -1334 ↗
  44. Xia Li, Minping Chen, Jianyun Nie, Zhenxing Liu, Ziheng Feng, and Yingdan Cai. 2018. Coherence-based automated essay scoring using self-attention. In Proceedings of the 17th China National Conference on Computational Linguistics, CCL 2018, and the 6th International Symposium on Natural Language Proc… — [doi:10.1007/978-3-030](https://doi.org/10.1007/978-3-030) find
  45. Mohsen Mesgar and Michael Strube. 2016. Lexical coherence graph modeling using word embeddings. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1414–1423. DOI: https://doi.org /10.18653/v1/N16-11… ↗
  46. Bin Liang, Hongcheng Li, Miaoqiang Su, Pan Bian, Xirong Li, and Wenchang Shi. 2018. Deep text classification can be fooled. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, pages 4208–4215. DOI: https://doi.org/10.24963/ijcai.2018/585 — [doi:10.24963/ijcai.2018/585](https://doi.org/10.24963/ijcai.2018/585) ↗
  47. Mohsen Mesgar and Michael Strube. 2018. A neural local coherence model for text quality assessment. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4328–4339. DOI: https://doi.org/10.18653/v1/D18 -1464 ↗
  48. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, cs.CL/1907.11692v1.
  49. Eleni Miltsakaki, Rashmi Prasad, Aravind Joshi, and Bonnie Webber. 2004. The Penn discourse treebank. In Proceedings of the Fourth International Conference on Language Resources and Evaluation.
  50. Farjana Sultana Mim, Naoya Inoue, Paul Reisert, Hiroki Ouchi, and Kentaro Inui. 2019. Unsupervised learning of discourse-aware text representation. In Proceedings of the 2019 Student Research Workshop, pages 93–104. find
  51. William C. Mann and Sandra A. Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text, 8(3):243–281. DOI: https://doi.org/10.1515/text.1.1988.8.3.243 ↗
  52. Ramesh Nallapati, Bowen Zhou, Cı́cero Nogueira dos Santos, Çaglar Gülçehre, and Bing Xiang. 2016. Abstractive text summarization using Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In sequence-to-sequence RNNs and beyond. In Proceedings of The 20th SIGNLL…
  53. Emily Pitler and Ani Nenkova. 2008. Revisiting readability: A unified framework for predicting text quality. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 186–195. DOI: https://doi.org/10.3115/1613715 .1613742 ↗
  54. Nicolas Papernot, Fartash Faghri, Nicholas Carlini, Ian Goodfellow, Reuben Feinman, Alexey Kurakin, Cihang Xie, Yash Sharma, Tom Brown, Aurko Roy, Alexander Matyasko, Vahid Behzadan, Karen Hambardzumyan, Zhishuai Zhang, Yi-Lin Juang, Zhi Li, Ryan Sheatsley, Abhibhav Garg, Jonathan Uesato, Willi Gier… find
  55. Shrimai Prabhumoye, Ruslan Salakhutdinov, and Alan W Black. 2020. Topological sort for sentence ordering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2783–2792. DOI: https://doi.org/10.18653/v1/2020 .acl-main.248 ↗
  56. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn discourse TreeBank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation.
  57. Cesc C. Park and Gunhee Kim. 2015. Expressing an image stream with a sequence of natural sentences. In Proceedings of Advances in Neural Information Processing Systems 28, pages 73–81. find
  58. Jan Wira Gotama Putra and Takenobu Tokunaga. 2017. Evaluating text coherence based on semantic similarity graph. In Proceedings of TextGraphs-11: the Workshop on Graph-based Methods for Natural Language Processing, pages 76–85. find
  59. Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2011. English gigaword fifth edition, linguistic data consortium. Google Scholar. find
  60. Suranjana Samanta and Sameep Mehta. 2017. Towards crafting text adversarial samples. CoRR, cs.LG/1707.02812v1. find
  61. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1532–1543. DOI: https://doi.org/10.3115/v1/D14-1162 ↗
  62. Motoki Sato, Jun Suzuki, Hiroyuki Shindo, and Yuji Matsumoto. 2018. Interpretable adversarial perturbation in input embedding space for text. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, pages 4323–4330. DOI: https://doi.org/10.24963/ijcai.2018/601 — [doi:10.24963/ijcai.2018/601](https://doi.org/10.24963/ijcai.2018/601) ↗
  63. Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018. Dissecting contextual word embeddings: Architecture and representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1499–1509. DOI: https://doi.org/10.18653/v1/D18-1179 ↗
  64. Swapna Somasundaran, Jill Burstein, and Martin Chodorow. 2014. Lexical chaining for measuring discourse coherence quality in test-taker essays. In Proceedings of the 25th International Conference on Computational Linguistics: Technical Papers, pages 950–961.
  65. Emily Pitler, Annie Louis, and Ani Nenkova. 2010. Automatic evaluation of linguistic quality in multi-document summarization. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 544–554. find
  66. Gongbo Tang, Mathias Müller, Annette Rios, and Rico Sennrich. 2018. Why self-attention? A targeted evaluation of neural machine translation architectures. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4263–4272. DOI: https://doi.org/10.18653/v1 /D1… find
  67. Deirdre Wilson and Dan Sperber. 2004. Relevance theory, The Handbook of Pragmatics, Blackwell, pages 607–632. find
  68. Yi Tay, Minh C. Phan, Luu Anh Tuan, and Siu Cheung Hui. 2018. SkipFlow: Incorporating neural coherence features for end-to-end automatic text scoring. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, pages 5948–5955. find
  69. Peng Xu, Hamidreza Saghir, Jin Sung Kang, Teng Long, Avishek Joey Bose, Yanshuai Cao, and Jackie Chi Kit Cheung. 2019. A cross-domain transferable neural coherence model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 678–687. find
  70. Dat Tien Nguyen and Shafiq Joty. 2017. A neural local coherence model. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1320–1330. DOI: https://doi.org/10.18653/v1/P17 -1121 ↗
  71. Puyudi Yang, Jianbo Chen, Cho-Jui Hsieh, Jane-Ling Wang, and Michael I. Jordan. 2020. Greedy attack and gumbel attack: Generating adversarial examples for discrete data. Journal of Machine Learning Research, 21:43:1–43:36. find
  72. Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. WikiQA: A challenge dataset for open-domain question answering. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2013–2018. DOI: https://doi.org/10.18653/v1/D15-1237 ↗
  73. Ke Tran, Arianna Bisazza, and Christof Monz. 2018. The importance of being recurrent for modeling hierarchical structure. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4731–4736. DOI: https://doi.org/10.18653/v1/D18-1503 ↗
  74. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Proceedings of the Thirty-third Conference on Neural Information Processing Systems, pages 5754–5764.
  75. Trieu H. Trinh and Quoc V. Le. 2018. A simple method for commonsense reasoning. CoRR, cs.AI/1806.02847v2. find
  76. Alex Warstadt, Amanpreet Singh, and Samuel Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7(0). DOI: https://doi.org/10.1162/tacl a 00290 ↗
  77. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369…
  78. Bonnie Webber. 2009. Genre distinctions for discourse in the Penn TreeBank. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 674–682. DOI: https://doi.org/10.3115/1690219 .169024… ↗
  79. Zeynep Yirmibeşoğlu and Gülşen Eryiğit. 2018. Detecting code-switching between Turkish-English language pair. In Proceedings of the 2018 EMNLP Workshop W-NUT: The 4th Workshop on Noisy User-generated Text, pages 110–115. DOI: https://doi.org/10.18653/v1/W18 -6115 ↗
  80. Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. 2018. What do RNN language models learn about filler–gap dependencies? In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 211–221. DOI: https://doi.org/10.18653/v1/W18 -5423 ↗
  81. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93–104. DOI: https://doi.org/10.18653/v1/D18 -1009 pages 4791–… ↗
  82. Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern R…
  83. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shen-2021-evaluating-document-coherence-modeling/