Elamur Research⌕
☰
home / papers / 2026 / Comhis at SemEval-2026 Task 4: Embedding-Space Adaptation and LLM-Assisted Inference for Narrative Similarity

Comhis at SemEval-2026 Task 4: Embedding-Space Adaptation and LLM-Assisted Inference for Narrative Similarity

Paper · 2026 · Ke Shu, Eetu Mäkelä and Mikko Tolonen

Summary

This paper describes the Comhis team's entry to SemEval-2026 Task 4. The task asks which of two candidate stories is narratively closer to an anchor story, judged by theme, course of action and outcome rather than word overlap. The system has two stages.

  • Track B (representation learning): Qwen3-Embedding-8B, served with vLLM, stays frozen. Only a linear projection layer on top of it is trained. This layer starts close to the identity matrix and is optimised with a cosine-based triplet loss and AdamW. Training uses synthetic comparison data plus the official example data. Hard examples are upweighted through weighted sampling; these are training/dev items the base embeddings ranked wrongly or with small margins.
  • Track A (pairwise judgment): cosine similarities from the adapted space, and their difference, are added to the prompt of an LLM, which makes the final choice. If the LLM output cannot be parsed, the system falls back to the raw embedding comparison.

Test-set accuracy came to 0.68 for Track A and 0.66 for Track B, both higher than the baselines. The work shows that adapting a frozen embedding model cheaply, then pairing it with an LLM, is an efficient route to modelling story-level similarity.

Key points

  • Official test results: Track A 0.68 accuracy (20th of 44 teams); Track B 0.66 (10th of 27).
  • Development results: full system 0.78 (Track A) and 0.82 (Track B), against baselines of 0.48 and 0.58.
  • Ablation on test (A/B): without synthetic data 0.57/0.55; without hard example mining 0.54/0.53. Removing hard example mining causes the larger drop.
  • Errors cluster where the two candidates' similarity scores are close. Recurring causes are lexical-overlap bias, stories that share a theme but end differently, and loosely structured episodic plots.
  • The LLM usually follows the embedding scores but sometimes overrides them, especially when themes match but outcomes differ. The authors read Track A's slight test advantage (0.68 vs 0.66) as a sign of this added value, although on dev Track B scored higher.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of HelsinkiHelsinki · Finland50 works · 5,318 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Ke Shu, Eetu Mäkelä and Mikko Tolonen. 2026. Comhis at SemEval-2026 Task 4: Embedding-Space Adaptation and LLM-Assisted Inference for Narrative Similarity. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.118

@inproceedings{shu2026comhis,
  title = {{Comhis at SemEval-2026 Task 4: Embedding-Space Adaptation and LLM-Assisted Inference for Narrative Similarity}},
  author = {Ke Shu, Eetu Mäkelä and Mikko Tolonen},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.118},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.118}
}

References

6 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Xi Chen, Ali Zeynali, Chico Camargo, Fabian Flöck, Devin Gaffney, Przemyslaw Grabowicz, Scott A. Hale, David Jurgens, and Mattia Samory. 2022. SemEval-2022 task 8: Multilingual news article similarity. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 109… find
  2. Gabriel de Souza P. Moreira, Radek Osmulski, Mengyao Xu, Ronay Ak, Benedikt Schifferer, and Even Oldridge. 2025. Nv-retriever: Improving text embedding models with effective hard-negative mining. Preprint, arXiv:2407.15831. — [arXiv:2407.15831](https://arxiv.org/abs/2407.15831) find
  3. Hans Ole Hatzel, Ekaterina Artemova, Haimo Stiemer, Evelyn Gius, and Chris Biemann. 2026. SemEval-2026 Task 4: Narrative similarity and narrative representation learning. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, CA, USA. Association for Comp… find
  4. Hans Ole Hatzel and Chris Biemann. 2024. Story embeddings — narrative-focused representations of fictional stories. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5931–5943, Miami, Florida, USA. Association for Computational Linguistics. find
  5. Anand Kumar and Hemanth Kumar. 2024. scaLAR SemEval-2024 task 1: Semantic textual relatednes for English. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 902–906, Mexico City, Mexico. Association for Computational Linguistics.
  6. Arjumand Younus and Muhammad Atif Qureshi. 2025. nlptuducd at SemEval-2025 task 10: Narrative classification as a retrieval task through story embeddings. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), pages 1742–1746, Vienna, Austria. Association for Comput… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shu-2026-comhis-semeval-2026-task-4/