Elamur Research⌕
☰
home / papers / 2026 / Explainable Semantic Textual Similarity via Dissimilar Span Detection

Explainable Semantic Textual Similarity via Dissimilar Span Detection

Summary

Semantic Textual Similarity (STS) systems usually return one number, such as a cosine score, which says nothing about where two texts diverge. The authors define a new task, Dissimilar Span Detection (DSD): given a text pair, locate the aligned spans that play the same role but carry different meaning. A pair may also contain no such spans.

Existing resources fit poorly. NLI contradiction pairs tend to differ too broadly, and the SemEval-2016 interpretable STS data offers few usable cases. The authors therefore built the Span Similarity Dataset (SSD). LLMs produced the annotations semi-automatically, and humans then checked them.

Several baselines were compared:

  • unsupervised attribution methods based on LIME and SHAP;
  • the authors' own SHAP-inspired method;
  • prompted LLMs;
  • fine-tuned encoder models.

LLMs scored best overall, and the supervised models gave the best trade-off between size and accuracy. Absolute scores stayed low across all methods, so DSD remains hard even in this simplified form. A follow-up experiment showed that DSD signals can improve paraphrase detection. This suggests span-level explanations can benefit STS-dependent applications such as fake news detection, clustering and parallel corpus mining.

Key points

  • SSD dataset: 1,000 annotated sentence pairs. A subset was manually re-annotated by 6 annotators to validate the semi-automatic labels. Each span pair is labelled 0 (dissimilar) or 1 (equivalent), with an additional sentence-level label.
  • Reuse of SemEval-2016 Task 2: only 109 of its 2,929 pairs were suitable for DSD, using mainly the OPPO label. This subset served as an extra evaluation set.
  • Models evaluated:
    • LLMs: GPT-4o, Llama 3.3 70B Instruct, Claude 3.5 Sonnet, DeepSeek R1.
    • Fine-tuned encoders: bert-base-uncased, roberta-base, distilbert-base-uncased, distilroberta-base.
    • Embedding models: all-MiniLM-L6-v2, all-mpnet-base-v2, text-embedding-3-large, text-embedding-004.
  • Licensing: code is released under GPL v3.0 and SSD under CC BY-SA 4.0, at dmlls.github.io/dissimilar-span-detection.
  • Missing figures: the provided extract omits the results tables, so exact DSD and paraphrase-detection scores need to be checked in the full paper.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4cewrdkiy6ph

@inproceedings{lozano2026explainable,
  title = {{Explainable Semantic Textual Similarity via Dissimilar Span Detection}},
  author = {Lozano, Diego Miguel and Dementieva, Daryna and Fraser, Alexander},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4cewrdkiy6ph},
  url = {https://doi.org/10.63317/4cewrdkiy6ph}
}

References

34 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Iñigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. 2015. SemEval-2015 task 2: Semantic textual similarity, English, Spanish and pilot on interpretabi… find
  2. Eneko Agirre, Aitor Gonzalez-Agirre, Iñigo Lopez-Gazpio, Montse Maritxalar, German Rigau, and Larraitz Uria. 2016. SemEval-2016 task 2: Interpretable semantic textual similarity. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 512– 524, San Diego, Calif… find
  3. Miriam Anschütz, Diego Miguel Lozano, and Georg Groh. 2023. This is not correct! Negation-aware Evaluation of Language Generation Systems. In Proceedings of the 16th International Natural Language Generation Conference, pages 163– 175, Prague, Czechia. Association for Computational Linguistics. find
  4. Michael Denkowski and Alon Lavie. 2014. Meteor Universal: Language Specific Translation Evaluation for Any Target Language. In Proceedings of the EACL 2014 Workshop on Statistical Machine Translation. find our card
  5. Sai Gurrapu, Ajay Kulkarni, Lifu Huang, Ismini Lourentzou, and Feras A. Batarseh. 2023. Rationalization for explainable NLP: a survey. Frontiers in Artificial Intelligence, 6. find
  6. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Onli… find our card
  7. Youngwoo Kim, Myungha Jang, and James Allan. 2020. Explaining Text Matching on Neural Natural Language Inference. ACM Trans. Inf. Syst., 38(4). find
  8. Jan-Christoph Klie, Bonnie Webber, and Iryna Gurevych. 2023. Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future. Computational Linguistics, 49(1):157–198. find our card
  9. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Proceedings of the 34th Internation… find
  10. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. ArXiv, abs/1907.11692. find
  11. I. Lopez-Gazpio, M. Maritxalar, A. Gonzalez-Agirre, G. Rigau, L. Uria, and E. Agirre. 2017. Interpretable semantic textual similarity: Finding and explaining differences between sentences. Knowledge-Based Systems, 119:186–199. find
  12. Scott M Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 4765–4774. Curran Associates, Inc. find our card
  13. Itzik Malkiel, Dvir Ginzburg, Oren Barkan, Avi Caciularu, Jonathan Weill, and Noam Koenigstein. 2022. Interpreting BERT-based Text Similarity via Activation and Saliency Maps. In Proceedings of the ACM Web Conference 2022, WWW ’22, page 3259–3268, New York, NY, USA. Association for Computing Machine… find
  14. Shervin Minaee, Nal Kalchbrenner, Erik Cambria, Narjes Nikzad, Meysam Chenaghlu, and Jianfeng Gao. 2021. Deep Learning–based Text Classification: A Comprehensive Review. ACM Comput. Surv., 54(3). find
  15. Lance Ramshaw and Mitch Marcus. 1995. Text Chunking using Transformation-Based Learning. In Third Workshop on Very Large Corpora. find
  16. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. find
  17. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why Should I Trust You?": Explaining the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pages 1135–1144. find our card
  18. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Anchors: high-precision model-agnostic explanations. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on E… find
  19. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108. find
  20. Aalok Sathe, Salar Ather, Tuan Manh Le, Nathan Perry, and Joonsuk Park. 2020. Automated fact-checking of claims from Wikipedia. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6874–6882, Marseille, France. European Language Resources Association. find
  21. L. S. Shapley. 1953. 17. A Value for n-Person Games, pages 307–318. Princeton University Press, Princeton. find
  22. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2019. Generating Token-Level Explanations for Natural Language Inference. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, V… find
  23. Thinh Hung Truong, Yulia Otmakhova, Timothy Baldwin, Trevor Cohn, Jey Han Lau, and Karin Verspoor. 2022. Not another negation benchmark: The NaN-NLI test suite for sub-clausal negation. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics… find
  24. Teemu Vahtola, Mathias Creutz, and Jörg Tiedemann. 2022. It is not easy to detect paraphrases: Analysing semantic similarity with antonyms and negation using the new SemAntoNeg benchmark. In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 24… find
  25. Chrysoula Zerva, Frédéric Blain, Ricardo Rei, Piyawat Lertvittayakumjorn, José G. C. de Souza, Steffen Eger, Diptesh Kanojia, Duarte Alves, Constantin Orăsan, Marina Fomicheva, André F. T. Martins, and Lucia Specia. 2022. Findings of the WMT 2022 Shared Task on Quality Estimation. In Proceedings of… find our card
  26. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. In International Conference on Learning Representations. find
  27. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023. A Survey of L… find our card
  28. Chao Zhou, Cheng Qiu, and Daniel E. Acuna. 2022. Paraphrase Identification with Deep Learning: A Review of Datasets and Methods. 9. find
  29. Language Resource References Agirre, Eneko and Gonzalez-Agirre, Aitor and Lopez-Gazpio, Iñigo and Maritxalar, Montse and Rigau, German and Uria, Larraitz. 2016. SemEval-2016 Task 2: Interpretable Semantic Textual Similarity . Association for Computational Linguistics. find
  30. Miriam Anschütz, Diego Miguel Lozano, and Georg Groh. 2023. This is not correct! Negation-aware Evaluation of Language Generation Systems. In Proceedings of the 16th International Natural Language Generation Conference, pages 163– 175, Prague, Czechia. Association for Computational Linguistics. find
  31. Bowman, Samuel R. and Angeli, Gabor and Potts, Christopher and Manning, Christopher D. 2015. A large annotated corpus for learning natural language inference. Association for Computational Linguistics. find
  32. Camburu, Oana-Maria and Rocktäschel, Tim and Lukasiewicz, Thomas and Blunsom, Phil. 2018. e-SNLI: Natural Language Inference with Natural Language Explanations. Curran Associates, Inc. find our card
  33. Vasile Rus and Mihai C. Lintean and Cristian Moldovan and William Baggett and Nobal B. Niraula and Brent and Morgan. 2012. The SIMI-LAR Corpus: A Resource To Foster The Qualitative Understanding of Semantic Similarity of Texts. European Language Resources Association (ELRA). find
  34. Zhang, Yuan and Baldridge, Jason and He, Luheng. 2019. PAWS: Paraphrase Adversaries from Word Scrambling. Google AI Language. Proc. of NAACL. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lozano-2026-explainable-semantic-textual-similarity-dissimilar/