Elamur ResearchβŒ•
☰
home / papers / 2022 / A Pragmatics-Centered Evaluation Framework for Natural Language Understanding

A Pragmatics-Centered Evaluation Framework for Natural Language Understanding

Summary

Widely used NLU benchmarks such as GLUE, SuperGLUE and SentEval mainly test literal, context-free meaning. High scores on them have encouraged claims that pretrained models learn universal text representations. The authors argue that this ignores pragmatics: speaker intent, implicature, illocutionary force and discourse structure, which matter in real applications like dialogue and document analysis.

To address this, they build PragmEval, an English benchmark that gathers 11 public classification datasets for single sentences or sentence pairs. The datasets cover discourse relations, dialog acts and similar phenomena, and none of them appears in existing suites. Some datasets were cleaned or rebalanced so that rare classes affect the score.

The paper also estimates human accuracy for several tasks. It evaluates BERT both directly and after an extra fine-tuning step on MNLI, DisSent, Discovery or PragmEval. MNLI, the most popular auxiliary task, does not give the best PragmEval results. This suggests that NLI-based training does not produce truly universal representations, which leaves room for better multi-task learning. The data and preprocessing code are publicly released.

Key points

  • PragmEval combines 11 pragmatics-oriented English datasets, including PDTB, GUM, STAC, SwitchBoard and MRDA. It is distributed via GitHub (synapse-developpement/PragmEval) and Hugging Face (pragmeval).
  • Only datasets missing from SentEval, GLUE and SuperGLUE were selected, so the benchmark complements those suites rather than duplicating them.
  • Label distributions were deliberately changed through duplicate removal and class subsampling. For example, more than 99% of GUM's original labels are "unattached", and about 80% of SwitchBoard dialog acts are statements.
  • BERT was compared with variants given intermediate fine-tuning on MNLI, DisSent, Discovery and PragmEval. Per-category transfer F1 scores are reported, and MNLI was not the strongest auxiliary signal.
  • Besides serving as an evaluation suite, PragmEval can be used as additional training data in a multi-task setup.

Our summary of the paper, in our own words β€” not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis Β· Speech act theory

Cited by

5 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 3 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{sileo2022pragmatics,
  title = {{A Pragmatics-Centered Evaluation Framework for Natural Language Understanding}},
  author = {Sileo, Damien and Muller, Philippe and Cruys, Tim Van de and Pradel, Camille},
  booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference},
  year = {2022},
  doi = {10.63317/2rsbmip74udb},
  url = {https://doi.org/10.63317/2rsbmip74udb}
}

References

61 references, 23 of them with a card of ours. β†— leads to the work, Β«findΒ» searches for one we have no id for yet.

  1. Agirre, E., Diab, M., Cer, D., and Gonzalez-Agirre, A. (2012). Semeval-2012 task 6: A pilot on semantic textual similarity. In Proceedings of the First Joint Conference on Lexical and Computational Semantics – Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation, pages 385–393. Association for Computational Linguistics. find our card
  2. Asher, N. and Lascarides, A. (2003). Logics of conversation. Cambridge University Press. find
  3. Asher, N., Hunter, J., Morey, M., Farah, B., and Afantenos, S. (2016). Discourse structure and dialogue acts in multiparty dialogue: the STAC corpus. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 2721–2727, PortoroΕΎ, Slovenia, May. European Language Resources Association (ELRA). find our card
  4. Austin, J. L. (1975). How to do things with words. Oxford university press. β†—
  5. Badene, S., Thompson, K., LorrΓ©, J.-P., and Asher, N. (2019). Data programming for learning discourse structure. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 640–645, Florence, Italy, July. Association for Computational Linguistics. find
  6. Bai, H. and Zhao, H. (2018). Deep enhanced representation for implicit discourse relation recognition. In Proceedings of the 27th International Conference on Computational Linguistics, pages 571–583, Santa Fe, New Mexico, USA, August. Association for Computational Linguistics. β†— our card
  7. Belinkov, Y. and Glass, J. (2019). Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49–72. β†— our card
  8. Bowman, S. R. (2016). Modeling natural language semantics in learned representations. Ph.D. thesis. find
  9. Buechel, S. and Hahn, U. (2017). EmoBank: Studying the impact of annotation perspective and representation format on dimensional emotion analysis. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 578–585, Valencia, Spain, April. Association for Computational Linguistics. find
  10. Carlile, W., Gurrapadi, N., Ke, Z., and Ng, V. (2018). Give me more feedback: Annotating argument persuasiveness and related attributes in student essays. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 621–631, Melbourne, Australia, July. Association for Computational Linguistics. find
  11. Cer, D., Yang, Y., yi Kong, S., Hua, N., Limtiaco, N., John, R. S., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., Sung, Y.-H., Strope, B., and Kurzweil, R. (2018). Universal sentence encoder. β†— our card
  12. Chen, M., Chu, Z., and Gimpel, K. (2019). Evaluation benchmarks and learning criteria for discourse-aware sentence representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 649–662, Hong Kong, China, November. Association for Computational Linguistics. β†—
  13. Conneau, A., Kiela, D., Schwenk, H., Barrault, L., and Bordes, A. (2017). Supervised Learning of Universal Sentence Representations from Natural Language Inference Data. Emnlp. find our card
  14. Conneau, A., Kruszewski, G., Lample, G., Barrault, L., and Baroni, M. (2018). What you can cram into a single vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2126–2136. Association for Computational Linguistics. β†— our card
  15. Core, M. G. and Allen, J. (1997). Coding dialogs with the DAMSL annotation scheme. In AAAI fall symposium on communicative action in humans and machines, volume 56, pages 28–35. Boston, MA. find
  16. de Marneffe, M.-C., Simons, M., and Tonhauser, J. (2019). The commitmentbank: Investigating projection in naturally occurring discourse. Proceedings of Sinn und Bedeutung, 23(2):107–124, Jul. β†— our card
  17. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). Association for Computational Linguistics. find
  18. Dolan, R. J. (2002). Emotion, cognition, and behavior. Science, 298(5596):1191–1194. β†—
  19. Ferreira, W. and Vlachos, A. (2016). Emergent: a novel data-set for stance classification. In HLT-NAACL. find
  20. Godfrey, J. J., Holliman, E. C., and McDaniel, J. (1992). Switchboard: Telephone speech corpus for research and development. In Proceedings of the 1992 IEEE International Conference on Acoustics, Speech and Signal Processing - Volume 1, ICASSP’92, pages 517–520, Washington, DC, USA. IEEE Computer Society. find
  21. Green, M. S. (2000). Illocutionary force and semantic content. Linguistics and Philosophy, 23(5):435–473. β†—
  22. Grice, H. P. (1975). Logic and conversation. In Peter Cole et al., editors, Syntax and Semantics: Vol. 3: Speech Acts, pages 41–58. Academic Press, New York. β†—
  23. Halliday, M. (1985). An Introduction to Functional Grammar. Edward Arnold Press, Baltimore. find
  24. Hovy, E. and Maier, E. (1992). Parsimonious or profligate: How many and which discourse structure relations? Technical Report RR-93-373, USC Information Sciences Institute. β†—
  25. Kiros, J. and Chan, W. (2018). InferLite: Simple universal sentence representations from natural language inference data. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4868–4874, Brussels, Belgium, October-November. Association for Computational Linguistics. find
  26. Kiros, R., Zhu, Y., Salakhutdinov, R. R., Zemel, R., Urtasun, R., Torralba, A., and Fidler, S. (2015). Skip-thought vectors. In Advances in neural information processing systems, pages 3294–3302. β†—
  27. Lahiri, S. (2015). SQUINKY! A Corpus of Sentence-level Formality, Informativeness, and Implicature. CoRR, abs/1506.02306. β†—
  28. LΓ€ubli, S., Sennrich, R., and Volk, M. (2018). Has machine translation achieved human parity? a case for document-level evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4791–4796, Brussels, Belgium, October-November. Association for Computational Linguistics. find our card
  29. Liu, X., He, P., Chen, W., and Gao, J. (2019). Improving multi-task deep neural networks via knowledge distillation for natural language understanding. arXiv preprint arXiv:1904.09482. β†—
  30. Mann, W. and Thompson, S. (1987). Rhetorical structure theory : a theory of text organization. Technical report, Information Science Institute. find
  31. McCann, B., Keskar, N. S., Xiong, C., and Socher, R. (2018). The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730. β†— our card
  32. Morey, M., Muller, P., and Asher, N. (2017). How much progress have we made on RST discourse parsing? a replication study of recent results on the RST-DT. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1319–1324, Copenhagen, Denmark, September. Association for Computational Linguistics. find our card
  33. Nangia, N. and Bowman, S. R. (2019). Human vs. muppet: A conservative estimate of human performance on the GLUE benchmark. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4566–4575, Florence, Italy, July. Association for Computational Linguistics. find
  34. Nie, A., Bennett, E. D., and Goodman, N. D. (2019). DisSent: Sentence Representation Learning from Explicit Discourse Relations. pages 4497–4510, July. find
  35. Oraby, S., Harrison, V., Reed, L., Hernandez, E., Riloff, E., and Walker, M. (2016). Creating and characterizing a diverse corpus of sarcasm in dialogue. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 31–41. Association for Computational Linguistics. find our card
  36. Park, J. and Cardie, C. (2014). Identifying appropriate support for propositions in online user comments. In Proceedings of the first workshop on argumentation mining, pages 29–38. find
  37. Pennington, J., Socher, R., and Manning, C. D. (2014). GloVe: Global Vectors for Word Representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1532–1543. find
  38. Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana, June. Association for Computational Linguistics. find
  39. Pfeffer, J. (1981). Understanding the role of power in decision making. Jay M. Shafritz y J. Steven Ott, Classics of Organization Theory, Wadsworth, pages 137–154. find
  40. Phang, J., FΓ©vry, T., and Bowman, S. R. (2018). Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. CoRR, abs/1811.01088. β†— our card
  41. Pitler, E., Raghupathy, M., Mehta, H., Nenkova, A., Lee, A., and Joshi, A. (2008). Easily identifiable discourse relations. In Coling 2008: Companion volume: Posters, pages 87–90. Coling 2008 Organizing Committee. find our card
  42. Poliak, A., Haldar, A., Rudinger, R., Hu, J. E., Pavlick, E., White, A. S., and Van Durme, B. (2018). Collecting diverse natural language inference problems for sentence representation evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 67–81. β†— our card
  43. Prasad, R., Dinesh, N., Lee, A., Miltsakaki, E., Robaldo, L., Joshi, A., and Webber, B. (2008). The penn discourse treebank 2.0. In Nicoletta Calzolari, et al., editors, Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), Marrakech, Morocco, may. European Language Resources Association (ELRA). http://www.lrec-conf.org/proceedings/lrec2008/. β†— our card
  44. Prasad, R., Riley, K. F., and Lee, A. (2014). Towards Full Text Shallow Discourse Relation Annotation : Experiments with Cross-Paragraph Implicit Relations in the PDTB. (2009). find our card
  45. Ribeiro, E., Ribeiro, R., and de Matos, D. M. (2015). The influence of context on dialogue act recognition. arXiv preprint arXiv:1506.00839. β†—
  46. Searle, J. R., Kiefer, F., Bierwisch, M., et al. (1980). Speech act theory and pragmatics, volume 10. Springer. find
  47. Shriberg, E., Dhillon, R., Bhagat, S., Ang, J., and Carvey, H. (2004). The icsi meeting recorder dialog act (mrda) corpus. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue at HLT-NAACL 2004. β†— our card
  48. Sileo, D., Van De Cruys, T., Pradel, C., and Muller, P. (2019a). Composition of sentence embeddings: Lessons from statistical relational learning. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), pages 33–43, Minneapolis, Minnesota, June. Association for Computational Linguistics. find
  49. Sileo, D., Van de Cruys, T., Pradel, C., and Muller, P. (2019b). Mining discourse markers for unsupervised sentence representation learning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). Association for Computational Linguistics. β†— our card
  50. Sileo, D., Van de Cruys, T., Pradel, C., and Muller, P. (2020). DiscSense: Automated semantic analysis of discourse markers. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 991–999, Marseille, France, May. European Language Resources Association. β†—
  51. Socher, R., Chen, D., Manning, C., Chen, D., and Ng, A. (2013). Reasoning With Neural Tensor Networks for Knowledge Base Completion. In Neural Information Processing Systems (2003), pages 926–934. find
  52. Subramanian, S., Trischler, A., Bengio, Y., and Pal, C. J. (2018). Learning general purpose distributed sentence representations via large scale multi-task learning. International Conference on Learning Representations. find
  53. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. (2018). GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium, November. Association for Computational Linguistics. β†— our card
  54. Wang, A., Hula, J., Xia, P., Pappagari, R., McCoy, R. T., Patel, R., Kim, N., Tenney, I., Huang, Y., Yu, K., Jin, S., Chen, B., Durme, B. V., Grave, E., Pavlick, E., and Bowman, S. R. (2019a). Can you tell me how to get past sesame street? sentence-level pretraining beyond language modeling. In ACL 2019. β†—
  55. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. (2019b). GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations. β†— our card
  56. Wang, A., Tenney, I. F., Pruksachatkun, Y., Yu, K., Hula, J., Xia, P., Pappagari, R., Jin, S., McCoy, R. T., Patel, R., Huang, Y., Phang, J., Grave, E., Liu, H., Kim, N., Htut, P. M., F’evry, T., Chen, B., Nangia, N., Mohananey, A., Kann, K., Bordia, S., Patry, N., Benton, D., Pavlick, E., and Bowman, S. R. (2019c). jiant 1.2: A software toolkit for research on general-purpose text understanding models. http://jiant.info/. β†—
  57. Warstadt, A., Singh, A., and Bowman, S. R. (2018). Neural network acceptability judgments. arXiv preprint arXiv:1805.12471. β†—
  58. Wieting, J., Bansal, M., Gimpel, K., and Livescu, K. (2015). Towards universal paraphrastic sentence embeddings. CoRR, abs/1511.08198. β†— our card
  59. Williams, A., Nangia, N., and Bowman, S. (2018). A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics. find
  60. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., and Le, Q. V. (2019). Xlnet: Generalized autoregressive pretraining for language understanding. β†— our card
  61. Zeldes, A. (2017). The GUM corpus: Creating multilayer resources in the classroom. Language Resources and Evaluation, 51(3):581–612. BERT B+MNLI B+DisSent B+Discovery B+PragmEval Support GUM.no_relation 48.9 GUM.circumstance 77.1 GUM.elaboration 41.5 STAC.no_relation 59.9 STAC.Comment 77.8 STAC.Question_answer_pair 79.1 SwitchBoard.Uninterpretable 86.0 SwitchBoard.Statement-non-opinion 72.0 SwitchBoard.Yes-No-Question 85.9 PDTB.Cause 55.2 PDTB.Restatement 40.4 PDTB.Conjunction 52.8 MRDA.Statement 51.2 MRDA.Defending/Explanation 52.8 MRDA.Expansions of y/n Answers 51.7 Table 5: Transfer F1 scores across the categories of PragmEval tasks; B(ERT)+X denotes BERT pretrained classification model after auxiliary finetuning phase on task X . β†—

↑ back to top

This page in Markdown β†’