Elamur Research⌕
☰
home / papers / 2026 / Similar, but why? A Toolkit for Explaining Text Similarity

Similar, but why? A Toolkit for Explaining Text Similarity

Paper · 2026 · Juri Opitz, Andrianos Michail, Lucas Moeller et al.

Summary

Similarity scores from embedding models are hard to justify because they come from the interaction of two inputs, not from one. Most existing interpretability libraries assume a single input and a single prediction. The authors present XPLAINSIM, an open Python package that offers three complementary ways to explain text similarity:

  • Token-pair attribution: an integrated-gradients-style decomposition for Siamese encoders, extended to multilingual and retrieval models, with optional alignment sparsification.
  • Space shaping: training embeddings whose subspaces correspond to user-defined semantic aspects.
  • Symbolic comparison: parsing texts into AMR graphs and comparing them with graph metrics.

Three pilot studies show what the package can do: cross-lingual token alignment, stop-word handling in IR models, and the difference between similarity and relatedness. In the third study, AMR-based aspect differences relate to human ratings differently on STS (similarity) and SICK (relatedness). This matters for search in sensitive fields such as law and medicine, where regulation increasingly demands explainable systems.

Key points

  • XPLAINSIM is released under GPLv3, installable via pip and imported as xplain. It builds on pytorch and sentence-transformers, plus amrlib and smatchpp for the symbolic module.
  • Attribution matrices can be discretized in two ways:
    • MaxAlign keeps only mutual best token links, favoring precision.
    • FlowAlign uses a Wasserstein/optimal-transport plan with threshold τ, which gives higher recall. The default τ = 0.029 was tuned on the alignment dev set.
  • In a demo with gte-multilingual-base on an English–German pair (cosine 0.82), the model aligns pairs such as dog–Hund and kitten–Katze. The retrieval study uses models such as msmarco-MiniLM-L12-cos-v5.
  • On STS, differences in polarity and named entities predict human similarity judgments, but they matter less for relatedness on SICK. Changes in patients track similarity, while agent structure matters more for relatedness.
  • Stated limitations:
    • Attribution is computationally expensive, so long texts are impractical without further adaptation.
    • Space shaping requires custom metrics and training.
    • AMR metrics correlate less with human ratings than large neural models do, partly because of parser noise.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of StuttgartStuttgart · Germany91 works · 46,376 citations🎓University of ZurichZurich · Switzerland78 works · 12,260 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 worksSemantic parsing & meaning representation · 185 works

Cite

Cite as Juri Opitz, Andrianos Michail, Lucas Moeller, Sebastian Padó and Simon Clematide. 2026. Similar, but why? A Toolkit for Explaining Text Similarity. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 3: System Demonstrations).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-demo.16

@inproceedings{opitz2026similar,
  title = {{Similar, but why? A Toolkit for Explaining Text Similarity}},
  author = {Juri Opitz, Andrianos Michail, Lucas Moeller, Sebastian Padó and Simon Clematide},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 3: System Demonstrations)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-demo.16},
  url = {https://doi.org/10.18653/v1/2026.eacl-demo.16}
}

References

42 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J Alammar. 2021. Ecco: An open source library for the explainability of transformer language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pages 249… find
  2. James F. Allen, Mary Swift, and Will de Beaumont. 2008. Deep semantic analysis of text. In Semantics in Text Processing. find
  3. STEP 2008 Conference Proceedings, pages 343–354. College Publications. find
  4. Giuseppe Attanasio, Eliana Pastor, Chiara Di Bonaventura, and Debora Nozza. 2023. ferret: a framework for benchmarking explainers on transformers. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 256–266, Dub… find
  5. Xuefeng Bai, Yulong Chen, and Yue Zhang. 2022. Graph pre-training for AMR parsing and generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6001–6015, Dublin, Ireland. Association for Computational Linguistics.
  6. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268.
  7. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discou…
  8. Rémi de Vergnette, Maxime Amblard, and Bruno Guillaume. 2025. Evaluation framework for layered meaning representation. In Proceedings of the Sixth International Workshop on Designing Meaning Representations, pages 38–48, Prague, Czechia. Association for Computational Lingustics. find
  9. Zi-Yi Dou and Graham Neubig. 2021. Word alignment by fine-tuning embeddings on parallel corpora. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2112–2128, Online. Association for Computational Linguistics. find
  10. EU. 2024. Article 86: Right to explanation of individual decision-making. find
  11. Jaden Fiotto-Kaufman, Alexander R. Loftus, Eric Todd, Jannik Brinkmann, Koyena Pal, Dmitrii Troitskii, Michael Ripa, Adam Belfki, Can Rager, Caden Juang, Aaron Mueller, Samuel Marks, Arnab Sen Sharma, Francesca Lucchetti, Nikhil Prakash, Carla Brodley, Arjun Guha, Jonathan Bell, Byron C. Wallace, an… find
  12. James Fodor, Simon De Deyne, and Shinsuke Suzuki. 2024. Compositionality and sentence meaning: Comparing semantic parsing and transformers on a challenging sentence similarity dataset. Computational Linguistics, pages 1–52.
  13. Sreyan Ghosh, Utkarsh Tyagi, Sonal Kumar, Chandra Kiran Evuru, Ramaneswaran S, S Sakshi, and Dinesh Manocha. 2024. ABEX: Data augmentation for low-resource NLU via expanding abstract descriptions. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: L… find
  14. Joseph D Janizek, Pascal Sturmfels, and Su-In Lee. 2021. Explaining explanations: Axiomatic feature interactions for deep networks. Journal of Machine Learning Research, 22(104):1–54. find
  15. Jelena Kallas, Svetla Peneva Koeva, Kristina Koppel, Simon Krek, Margit Langemets, Veronika Lipp, Sanni Nimb, Sussi Olsen, Bolette Sandford Pedersen, Valeria Quochi, Ana Salgado, László Simon, Carole Tiberius, Rafael-J. Ureña-Ruiz, and Roberto Navigli. 2023. XL-WA: a gold evaluation benchmark for wo… find
  16. Chathuri Jayaweera, Sangpil Youm, and Bonnie J Dorr. 2024. AMREx: AMR for explainable fact verification. In Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER), pages 234– 244, Miami, Florida, USA. Association for Computational Linguistics. find
  17. Lucas Moeller, Dmitry Nikolaev, and Sebastian Padó. 2023. An attribution method for Siamese encoders. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15818–15827, Singapore. Association for Computational Linguistics. find
  18. Zoher Kachwala, Jisun An, Haewoon Kwak, and Filippo Menczer. 2024. REMATCH: Robust and efficient matching of local knowledge graphs to improve structural and semantic similarity. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1018–1028, Mexico City, Mexico. Associati… find
  19. Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz-Richardson. 2020. Captum: A unified and generic model interpretability library for pytorch. Preprint, arXiv:2009.07896. — [arXiv:2009.07896](https://arxiv.org/abs/2009.07896) find
  20. Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015. From word embeddings to document distances. In International conference on machine learning, pages 957–966. PMLR. find
  21. Paul Landes and Barbara Di Eugenio. 2024. CALAMR: Component ALignment for Abstract Meaning Representation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 2622–2637, Torino, Italia. ELRA and ICCL. find
  22. Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281.
  23. Emma Manning and Nathan Schneider. 2021. Referenceless parsing-based evaluation of AMR-to-English generation. In Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems, pages 114–122, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  24. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC‘14), pages…
  25. Federico Martelli, Andrei Stefan Bejgu, Cesare Campagnano, Jaka Cibej, Rute Costa, Apolonija Gantar, Lucas Moeller, Dmitry Nikolaev, and Sebastian Padó. 2024. Approximate attributions for off-the-shelf Siamese transformers. In Proceedings of the 18th Conference of the European Chapter of the Associa… find
  26. Almuth Müller and Achim Kuwertz. 2022. Evaluation of a semantic search approach based on amr for information retrieval in image exploitation. In 2022 Sensor Data Fusion: Trends, Solutions, Applications (SDF), pages 1–6. IEEE. find
  27. Viswanath Nagarajan and Maxim Sviridenko. 2009. On the maximum quadratic assignment problem. Mathematics of Operations Research, 34(4):859–868. find
  28. Juri Opitz. 2023. SMATCH++: Standardized and extended evaluation of semantic graphs. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1595–1607, Dubrovnik, Croatia. Association for Computational Linguistics.
  29. Juri Opitz and Anette Frank. 2021. Towards a decomposable metric for explainable evaluation of text generation from AMR. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1504–1518, Online. Association for Computational…
  30. Juri Opitz and Anette Frank. 2022. SBERT studies meaning representations: Decomposing sentence embeddings into explainable semantic features. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference… find
  31. Juri Opitz, Lucas Moeller, Andrianos Michail, Sebastian Padó, and Simon Clematide. 2025. Interpretable text embeddings and text similarity explanation: A survey. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 22314–22330, Suzhou, China. Association f… find
  32. Juri Opitz, Shira Wein, Julius Steen, Anette Frank, and Nathan Schneider. 2023. AMR4NLI: Interpretable and robust NLI measures from semantic graphs. In Proceedings of the 15th International Conference on Computational Semantics, pages 275–283, Nancy, France. Association for Computational Linguistics… find
  33. Jinwoo Park, Hosoo Shin, Dahee Jeong, and Junyeong Kim. 2024. Improving the representation of sentences with reinforcement learning and amr graph. In 2024 IEEE International Conference on Consumer Electronics (ICCE), pages 1–4. find
  34. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,…
  35. Zacchary Sadeddine, Juri Opitz, and Fabian Suchanek. 2024. A survey of meaning representations – from theory to practical utility. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)… find
  36. Haibo Sun and Nianwen Xue. 2024. Anchor and broadcast: An efficient concept alignment approach for evaluation of semantic graphs. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 1052–1062, Torino, Ita… find
  37. Mukund Sundararajan, Kedar Dhamdhere, and Ashish Agarwal. 2020. The shapley taylor interaction index. In International conference on machine learning, pages 9259–9268. PMLR. find
  38. Ramya Keerthy Thatikonda, Wray Buntine, and Ehsan Shareghi. 2025. Assessing the sensitivity and alignment of FOL closeness metrics. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 16775–16785, Suzhou, China. Association for Computational Linguistics. find
  39. Shira Wein and Juri Opitz. 2024. A survey of AMR applications. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6856–6875, Miami, Florida, USA. Association for Computational Linguistics. find
  40. Shira Wein and Nathan Schneider. 2024. Assessing the cross-linguistic utility of abstract meaning representation. Computational Linguistics, 50(2):419–473.
  41. Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. 2023. MIRACL: A multilingual retrieval dataset covering 18 diverse languages. Transactions Short Name Text Embedding Model HF ID Base Model HF ID XLM-R M-M…
  42. FacebookAI/xlm-roberta-base sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 microsoft/Multilingual-MiniLM-L12-H384 intfloat/multilingual-e5-base FacebookAI/xlm-roberta-base Alibaba-NLP/gte-multilingual-base Alibaba-NLP/gte-multilingual-mlm-base *Table 3: Model Nomenclature: Mapping of sh… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/opitz-2026-similar-but-why-toolkit-explaining/