Elamur Research⌕
☰
home / papers / 2022 / Towards Human Evaluation of Mutual Understanding in Human-Computer Spontaneous Conversation: An Empirical Study of Word Sense Disambiguation for Naturalistic Social Dialogs in American English

Towards Human Evaluation of Mutual Understanding in Human-Computer Spontaneous Conversation: An Empirical Study of Word Sense Disambiguation for Naturalistic Social Dialogs in American English

Paper · 2022 · Alex Lưu

Summary

Human evaluation of social chatbots usually rates only how good the generated replies look. It rarely checks whether the system actually understood its human partner. Lưu argues that word sense disambiguation (WSD) should be a core, human-inspectable part of such evaluation. If a system exposes which WordNet sense it chose for an ambiguous word, an evaluator can trace a bad reply back to a misreading instead of guessing.

To test feasibility, the author used NEWT-SBCSAE, a small public corpus of casual face-to-face American English dialog. Ambiguous words were tagged with WordNet 3.0 senses by two systems:

  • a SOTA model, the back end of AMuSE-WSD;
  • a simple baseline that compares contextual embeddings of a word with embeddings of its sense glosses.

Two trained annotators then judged the outputs and supplied correct senses where needed. WordNet covered about 96% of the meanings. The SOTA model reached roughly 74% accuracy, below its score on written benchmarks, and the DistilBERT baseline reached 36%. The study shows the evaluation idea is practical for English conversation, and it supplies an annotated conversational WSD resource.

Key points

  • Data and preprocessing. NEWT-SBCSAE consists of seven 15-minute excerpts taken out of SBCSAE, the Santa Barbara spoken American English corpus, divided into 3253 turn-constructional units (TCUs). It was preprocessed with spaCy v2.3.5 (small English model). A token counted as ambiguous if its POS exists in WordNet and it has more than one synset.
  • Models.
    • SOTA model: Conia and Navigli (2021), the engine behind AMuSE-WSD (AW). It uses frozen BERT large-cased plus a classifier, is trained on SemCor and WordNet glosses, and scores about 80% on the Raganato et al. unified English benchmark.
    • Baseline: DistilBERT similarity between a word and its sense definitions, accessed via spacy-wordnet and spacy-transformers.
  • Annotation scale and agreement. 6350 ambiguous tokens were annotated over two rounds. In round 1, both annotators labelled the same dialog, giving raw agreement 0.750 and kappa 0.660 (counted as substantial).
  • Coverage and accuracy.
    • WordNet had a suitable sense for 5227 of 5427 judged tokens (96.3%).
    • AW accuracy was 73.7% overall and fairly even across POS.
    • The DistilBERT baseline reached 36.0%, with verbs hardest (26.1%). Verbs average 15.5 senses per token, against 9.9 for all tokens.
  • Swapping the baseline's transformer. All alternatives did worse than DistilBERT: BERT 34.2%, RoBERTa 30.2%, XLNet 18.5%.

Interesting ideas

  • I1. Evaluate understanding by asking the system to reveal intermediate interpretive choices, such as the word sense it picked, so human judges inspect the reasoning trail rather than rating the final text on vague scales.
  • I2. The judgment labels separate four failure types: wrong sense, wrong POS on a content word, wrong POS on a function word, and no sense in the inventory at all. This keeps upstream tagging mistakes from being counted as disambiguation mistakes.
  • I3. Before measuring a model, first ask how often the sense inventory itself has a fitting entry for real text, which caps any WSD score.
  • I4. A transparent baseline ranks a word's WordNet glosses by embedding similarity to the word in context, and can sit inside an existing pipeline instead of acting as an opaque end-to-end component.
  • I5. Two systems are judged in one annotation pass, with a shared label for "both correct", which halves human effort. The author also warns that a fixed display order can bias judges toward the system they come to trust.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): evaluation sets could score the sense, frame and entity choices behind each answer, so a wrong answer is attributed to a specific stage.
  • I2 (tokens): a similar error taxonomy would show how often tagger or lemma errors, not sense logic, cause downstream failures in en.
  • I3 (world): sampling our texts to estimate how often WordNet or FrameNet lacks a suitable sense or frame would show where compiled knowledge needs extending.
  • I4 (world): gloss-similarity ranking could give each linked entity an explainable candidate sense list, with a recorded score for every choice.
  • I5 (learn): when comparing witness parsers or teacher tiers, randomize the order outputs are shown in, so judges do not drift toward a favourite.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Brandeis UniversityWaltham · United States34 works · 5,637 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 4Research subjects and languages the work is about.English · 2,379 worksEvaluation & benchmarks · 1,341 worksDialogue systems · 654 worksWord sense disambiguation · 87 works

Cite

Cite as Alex Lưu. 2022. Towards Human Evaluation of Mutual Understanding in Human-Computer Spontaneous Conversation: An Empirical Study of Word Sense Disambiguation for Naturalistic Social Dialogs in American English. Proceedings of the 2nd Workshop on Human Evaluation of NLP Systems (HumEval).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.humeval-1.10

@inproceedings{luu2022human,
  title = {{Towards Human Evaluation of Mutual Understanding in Human-Computer Spontaneous Conversation: An Empirical Study of Word Sense Disambiguation for Naturalistic Social Dialogs in American English}},
  author = {Alex Lưu},
  booktitle = {Proceedings of the 2nd Workshop on Human Evaluation of NLP Systems (HumEval)},
  year = {2022},
  doi = {10.18653/v1/2022.humeval-1.10},
  url = {https://doi.org/10.18653/v1/2022.humeval-1.10}
}

References

23 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Michele Bevilacqua, Tommaso Pasini, Alessandro Raganato, and Roberto Navigli. 2021. Recent trends in word sense disambiguation: A survey. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 4330–4338. International Joint Conferences on Artificial Intelligence Organization. Survey Track. ↗
  2. Serge Bibauw, Thomas François, and Piet Desmet. 2019. Discussing with a computer to practice a foreign language: research synthesis and conceptual framework of dialogue-based CALL. Computer Assisted Language Learning, 32(8):827–877. Publisher: Routledge _eprint: https://doi.org/10.1080/09588221.2018.1535508. ↗
  3. Herbert H. Clark. 1996. Using Language. ‘Using’ Linguistic Books. Cambridge University Press, Cambridge. find
  4. Simone Conia and Roberto Navigli. 2021. Framing word sense disambiguation as a multi-label problem for model-agnostic knowledge integration. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3269–3275, Online. Association for Computational Linguistics. ↗
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  6. A. Seza Doğruöz and Gabriel Skantze. 2021. How “open” are the conversations with open-domain chatbots? a proposal for speech event based evaluation. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 392–402, Singapore and Online. Association for Computational Linguistics. find
  7. John W Du Bois, Wallace L Chafe, Charles Meyer, Sandra A Thompson, and Nii Martey. 2000. Santa Barbara corpus of spoken American English. CD-ROM. Philadelphia: Linguistic Data Consortium. find
  8. Christiane Fellbaum. 2010. WordNet. In Roberto Poli, Michael Healy, and Achilles Kameas, editors, Theory and Applications of Ontology: Computer Applications, pages 231–243. Springer Netherlands, Dordrecht. find
  9. Sarah E. Finch and Jinho D. Choi. 2020. Towards unified dialogue system evaluation: A comprehensive analysis of current evaluation protocols. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 236– 245, 1st virtual meeting. Association for Computational Linguistics. find
  10. Abed Alhakim Freihat, Fausto Giunchiglia, and Biswanath Dutta. 2016. A taxonomic classification of WordNet polysemy types. In Proceedings of the 8th Global WordNet Conference (GWC), pages 106– 114, Bucharest, Romania. Global Wordnet Association. find
  11. Amir Ahmad Habibi, Bradley Hauer, and Grzegorz Kondrak. 2021. Homonymy and polysemy detection with multilingual information. In Proceedings of the 11th Global Wordnet Conference, pages 26–35, University of South Africa (UNISA). Global Wordnet Association. find
  12. Mika Hämäläinen and Khalid Alnajjar. 2021. The great misalignment problem in human evaluation of NLP methods. In Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval), pages 69–74, Online. Association for Computational Linguistics. ↗
  13. Hendrik Heuer and Daniel Buschek. 2021. Methods for the design and evaluation of HCI+NLP systems. In Proceedings of the First Workshop on Bridging Human–Computer Interaction and Natural Language Processing, pages 28–33, Online. Association for Computational Linguistics. find
  14. Minlie Huang, Xiaoyan Zhu, and Jianfeng Gao. 2020. Challenges in Building Intelligent Open-domain Dialog Systems. ACM Transactions on Information Systems, 38(3):21:1–21:32. ↗
  15. Arkadiusz Janz and Marek Maziarz. 2021. Discriminating homonymy from polysemy in wordnets: English, Spanish and Polish nouns. In Proceedings of the 11th Global Wordnet Conference, pages 53–62, University of South Africa (UNISA). Global Wordnet Association. find
  16. J. Richard Landis and Gary G. Koch. 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics, 33(1):159. ↗
  17. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692 [cs]. ArXiv: 1907.11692. ↗
  18. Alex Lưu and Sophia A. Malamud. 2020. Annotating coherence relations for studying topic transitions in social talk. In Proceedings of the 14th Linguistic Annotation Workshop, pages 174–179, Barcelona, Spain. Association for Computational Linguistics. find
  19. Julie Mennes and Stephan van der Waart van Gulik. 2020. A critical analysis and explication of word sense disambiguation as approached by natural language processing. Lingua, 243:102896. ↗
  20. George A. Miller, Martin Chodorow, Shari Landes, Claudia Leacock, and Robert G. Thomas. 1994. Using a semantic concordance for sense identification. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994.
  21. Dieke Oele and Gertjan van Noord. 2018. Simple embedding-based word sense disambiguation. In Proceedings of the 9th Global Wordnet Conference, pages 259–265, Nanyang Technological University (NTU), Singapore. Global Wordnet Association. find
  22. Maarten van Gompel, Ko van der Sloot, Martin Reynaert, and Antal van den Bosch. 2017. FoLiA in Practice: The Infrastructure of a Linguistic Annotation Format, pages 71–82. Ubiquity Press. find
  23. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized Autoregressive Pretraining for Language Understanding. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/luu-2022-human-evaluation-mutual-understanding-human/