Elamur Research⌕
☰
home / papers / 2026 / Position: A Semiotic-Hermeneutic Approach to Evaluating Meaning in LLM Summaries via the Inductive Conceptual Rating Metric

Position: A Semiotic-Hermeneutic Approach to Evaluating Meaning in LLM Summaries via the Inductive Conceptual Rating Metric

Paper · 2026 · Natalie Perez, Sreyoshi Bhaduri and Aman Chadha

Summary

The paper questions whether standard automatic metrics, which treat words as stable units, can tell us if an LLM summary actually preserves the meaning of its source. Drawing on Saussure's semiotics and Gadamer's hermeneutics, the authors argue that meaning depends on context and interpretation. They propose the Inductive Conceptual Rating (ICR) metric. First, human coders run a blind Reflective Thematic Analysis (RTA, Braun and Clarke's six phases) on the reference data to build an interpretive baseline of themes. Then Inductive Content Analysis (ICA, Zhang and Wildemuth's eight steps) checks which of those concepts LLM outputs retain, weaken, or omit, and the result is turned into a coverage score. The case study compares thematic summaries of open-text survey answers to a single question, written by humans and by two LLMs, across five datasets. The models score well on lexical similarity metrics but cover far fewer of the human-identified themes. The authors conclude that surface fluency can conceal semantic misalignment, so interpretive evaluation needs human baselines.

Key points

  • Data and models: five datasets of 50, 100, 200, 400 and 800 survey responses. Human analysts are compared with Sonnet 3.5 and Nova Pro using cosine similarity, recall, precision, F1 and ICR.
  • Main gap: human ICR stays high, at about 0.93 on average (range 0.86–1.00). LLM ICR ranges from 0.35 to 0.76, even where the models' cosine similarity and F1 match or beat the human scores.
  • Effect of dataset size: the best LLM result appears at N = 800, where Nova Pro reaches ICR 0.76. Larger inputs help, but they do not bring models to human-level interpretation. Newer or more sophisticated models were not reliably better.
  • Scope: ICR is intended for abstractive summarization, thematic synthesis, and reference-based open generation. It is not meant for factual QA or creative generation.
  • Limitations:
    • The study covers a single survey question and only two LLMs.
    • The metric relies on presence or absence of concepts and does not capture discourse-level meaning.
    • Its validity depends on the coders' expertise and positionality.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Hawaiʻi at MānoaHonolulu · United States4 works · 20 citations

Also on the byline: University of Hawaii System.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Natalie Perez, Sreyoshi Bhaduri and Aman Chadha. 2026. Position: A Semiotic-Hermeneutic Approach to Evaluating Meaning in LLM Summaries via the Inductive Conceptual Rating Metric. Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.gem-main.81

@inproceedings{perez2026position,
  title = {{Position: A Semiotic-Hermeneutic Approach to Evaluating Meaning in LLM Summaries via the Inductive Conceptual Rating Metric}},
  author = {Natalie Perez, Sreyoshi Bhaduri and Aman Chadha},
  booktitle = {Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)},
  year = {2026},
  doi = {10.18653/v1/2026.gem-main.81},
  url = {https://doi.org/10.18653/v1/2026.gem-main.81}
}

References

21 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Alizadeh and A. Seilsepour. 2025. A novel self-supervised sentiment classification approach using semantic labeling based on contextual embeddings. Multimedia Tools and Applications, 84(12):10195– 10220. find
  2. M. Apidianaki. 2023. From word types to tokens and back: A survey of approaches to word meaning representation and interpretation. Computational Linguistics, 49(2):465–523.
  3. A. Arseniev-Koehler. 2024. Theoretical foundations and limits of word embeddings: what types of meaning can they capture? Sociological Methods & Research, 53(4):1753–1793. find
  4. R. Bedemariam, N. Perez, S. Bhaduri, S. Kapoor, A. Gil, E. Conjar, and N. Nayyar. 2025. Potential and perils of large language models as judges of unstructured textual data. arXiv preprint arXiv:2501.08167. — [arXiv:2501.08167](https://arxiv.org/abs/2501.08167) find
  5. S. Bhaduri, S. Kapoor, A. Gil, A. Mittal, and R. Mulkar. 2024. Reconciling methodological paradigms: Employing large language models as novice qualitative research assistants in talent management research. arXiv preprint, abs/2408.11043. — [arXiv:2408.11043](https://arxiv.org/abs/2408.11043) find
  6. A. D. Bhargavi. 2025. Comparative study of static and contextual text vectorization for sentiment analysis. International Journal for Research in Applied Science & Engineering Technology. find
  7. M. Borgstede and M. Scholz. 2021. Quantitative and qualitative approaches to generalization and replication – a representational perspective. Frontiers in Psychology, 12:605823. find
  8. V. Braun and V. Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2):77–101. find
  9. E. N. Purba and B. A. R. Damanik. 2025. Polysemy and homonymy in semantic interpretation. Young Journal of Social Sciences and Humanities, 1(3):101–109. find
  10. V. Braun and V. Clarke. 2022. Conceptual and design thinking for thematic analysis. Qualitative Psychology, 9(1):3. find
  11. F. De Saussure. 1916. Nature of the linguistic sign. Bedford/St. Martin’s Press. find
  12. J. W. Creswell and J. D. Creswell. 2017. Research design: Qualitative, quantitative, and mixed methods approaches. Sage Publications. find
  13. S. Shan. 2025. Towards reflexive ai: A comprehensive exploration of enhancing social science research through nlp. In Future of Information and Communication Conference, pages 765–792. Springer Nature Switzerland. find
  14. J. W. Creswell and C. N. Poth. 2016. Qualitative inquiry and research design: Choosing among five approaches. Sage Publications. find
  15. E. Fitkov-Norris, N. Kocheva, and N. Bhimireddy. 2023. Are we there yet? thematic analysis, nlp and machine learning for academic research: a comparative review. In FBSS Research Conference 2023. find
  16. H. G. Gadamer. 1975. Hermeneutics and social science. Cultural Hermeneutics, 2(4):307–316. find
  17. R. Gangundi and R. Sridhar. 2025. Rbca-ets: enhancing extractive text summarization with contextual embedding and word-level attention. International Journal of Information Technology, 17(2):1127–1135. find
  18. J. Grindrod. 2024. Transformers, contextualism, and polysemy. arXiv preprint arXiv:2404.09577. — [arXiv:2404.09577](https://arxiv.org/abs/2404.09577) find
  19. T. C. Guetterman, T. Chang, M. DeJonckheere, T. Basu, E. Scruggs, and V. V. Vydiswaran. 2018. Augmenting qualitative text analysis with natural language processing: methodological study. Journal of Medical Internet Research, 20(6):e231. find
  20. F. Viegas, A. Pereira, W. Cunha, C. França, C. Andrade, E. Tuler, and M. A. Gonçalves. 2025. Exploiting contextual embeddings in hierarchical topic modeling and investigating the limits of the current evaluation metrics. Computational Linguistics, pages 1–41. find
  21. Y. Zhang and B. M. Wildemuth. 2009. Qualitative analysis of content. In Applications of social research methods to questions in information and library science, volume 308(319), pages 1–12. Libraries Unltd Inc. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/perez-2026-position-semiotic-hermeneutic-approach-evaluating/