Elamur Research⌕
☰
home / papers / 2023 / Linguistic ambiguity analysis in ChatGPT

Linguistic ambiguity analysis in ChatGPT

Paper · 2023 · Miguel Ortega-Martin, Oscar Garcia-Sierra, Alfonso Ardoiz et al.

Summary

Ambiguity at many linguistic levels remains a core obstacle for NLP systems. This paper reviews three kinds: lexical (homonymy and polysemy), syntactic (structural attachment) and semantic (mainly pronoun co-reference). It explains how contextual embeddings and attention have eased some of these problems, then tests how ChatGPT copes. The only public benchmark, BIG-bench Sentence-Ambiguity, targets discourse-level phenomena, so the authors built their own small English test sets. They prompted the model to say whether each sentence is ambiguous, then followed up with targeted questions about word meanings, definition classification, pronoun reference and generating an ambiguous sentence. Overall detection reached accuracy of about 0.61 and F1 of 0.48. ChatGPT resolves co-reference ambiguity well and often names missing context as the cause. However, it over-detects polysemy and needs careful prompting for syntactic cases. It also lets gender stereotypes override unambiguous pronouns and contradicts itself about sentences it generated. The work matters as an early, reproducible snapshot of LLM behaviour on ambiguity, with prompts and outputs in the appendices.

Key points

  • Detection scores over all ambiguity types are accuracy 0.6061 and F1 0.48. For homonymy, false positives dominate (7 versus 3 TP, 2 TN, 1 FN), mostly because the model flags polysemy inside homonyms.
  • For polysemy, ChatGPT correctly rejected every non-ambiguous example but missed the ambiguous ones. In co-reference detection it scored perfectly.
  • In the paper's motivation, the spaCy model en_core_web_lg misparses a sentence in which John is the one holding a telescope, wrongly linking the "telescope" to Mary, the subject.
  • BIG-bench Sentence-Ambiguity was set aside because it covers discourse-level ambiguity. The basic prompt was: Is the sentence "[sentence]" ambiguous?
  • The authors list several limitations: the study covers English only, ChatGPT keeps changing over time, and leftover conversation context sometimes pushed the model to agree with the user, which forced new sessions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Universidad Complutense de MadridMadrid · Spain9 works · 63 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Lexical semantics & word embeddings · 63 works

Cite

Cite as Miguel Ortega-Martin, Oscar Garcia-Sierra, Alfonso Ardoiz, Jorge Alvarez, Juan Carlos Armenteros and Adrian Alonso. 2023. Linguistic ambiguity analysis in ChatGPT. arXiv:2302.06426.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2302.06426 · arXiv 2302.06426

@misc{ortegamartin2023linguistic,
  title = {{Linguistic ambiguity analysis in ChatGPT}},
  author = {Miguel Ortega-Martin, Oscar Garcia-Sierra, Alfonso Ardoiz, Jorge Alvarez, Juan Carlos Armenteros and Adrian Alonso},
  year = {2023},
  doi = {10.48550/arxiv.2302.06426},
  eprint = {2302.06426},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2302.06426}
}

References

12 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. \#\# 3. Going to the right is the right choice. find
  2. Muneera Bano. 2015. Addressing the challenges of requirements ambiguity: A review of empirical literature. In 2015 IEEE Fifth International Workshop on Empirical Requirements Engineering (EmpiRE), pages 21–24. IEEE. find
  3. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of" bias" in nlp. arXiv preprint arXiv:2005.14050.
  4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, and Amanda Askell. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  5. Dan Jurafsky and James H. Martin. 2020. Speech and language processing (3rd draft ed.). Stanford Univ. find
  6. Shaidah Jusoh. 2018. A study on NLP appliactions and ambiguity problems. Journal of Theoretical & Applied Information Technology, 96(6). ISBN: 1992-8645. find
  7. Qi Liu, Matt J Kusner, and Phil Blunsom. 2020. A survey on contextual embeddings. arXiv preprint arXiv:2003.07278. find
  8. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
  9. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, and Alex Ray. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155.
  10. Aarohi Srivastava et al. 2022. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. ArXiv:2206.04615 [cs, stat].
  11. Huiqiang Xie, Zhijin Qin, Geoffrey Ye Li, and Biing-Hwang Juang. 2021. Deep learning enabled semantic communication systems. IEEE Transactions on Signal Processing, 69:2663–2675. find
  12. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. arXiv preprint arXiv:1804.06876.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ortega-martin-2023-linguistic-ambiguity-analysis-chatgpt/