Elamur Research⌕
☰
home / papers / 2026 / JCT at SemEval-2026 Task 4: A Multi-Method Approach to Narrative Story Similarity

JCT at SemEval-2026 Task 4: A Multi-Method Approach to Narrative Story Similarity

Paper · 2026 · Dvori Rosenfeld, Rinat Walles and Chaya Liebeskind

Summary

The paper describes the JCT team's system for SemEval-2026 Task 4 (Track A). Given an anchor story and two candidates, a system must pick the candidate whose narrative is closer. Closeness is judged by abstract theme, course of action and outcomes, not by setting, character names or writing style. The authors compare three complementary strategies:

  • Fine-tuning: they adapt the uhhlt/story-emb embedding model with QLoRA and Triplet Margin Loss, using the official data plus an organizer-provided synthetic set of LLM-generated triples.
  • Distill-then-Embed: Gemini 1.5 Flash rewrites each story into a neutral "narrative core", which is then embedded with text-embedding-004 and compared by cosine similarity.
  • Zero-shot ensemble: Gemini, GPT and Claude models are prompted directly and their answers combined by majority voting.

The top score, 0.75 accuracy, was achieved by jointly using GPT-5, Claude Opus 4.5 and Gemini 2.5, placing 7th out of 47 teams. The locally run fine-tuned model reached 0.69. This shows that a cheap, open-weight, privacy-preserving option comes reasonably close to costly proprietary APIs.

Key points

  • Accuracy on the official test set:
    • GPT-4o-mini zero-shot baseline: 0.55
    • Distill-then-Embed: 0.66
    • QLoRA story-emb: 0.69
    • Gemini 2.5 Flash alone: 0.71
    • Gemini 2.5 + GPT-4o + Claude Sonnet 4.5 ensemble: 0.74
    • Gemini 2.5 + GPT-5 + Claude Opus 4.5 ensemble: 0.75
  • Fine-tuning setup:
    • 1,900 synthetic triples
    • 4-bit NF4 quantization; LoRA r=16, α=32, dropout 0.05
    • Triplet margin 0.5; AdamW, learning rate 1e-5, 5 epochs
    • Single A100 GPU
  • Ensemble robustness: hard voting skips models that fail to answer (e.g., API errors or safety filters). Ties are broken by a fixed priority ranking of models.
  • Error analysis: over 400 test pairs, the best ensemble made 65 false negatives versus 34 false positives. This means it tends to answer "Text A is not closer" when uncertain.
  • Code availability: code, data splits and prompts are published on GitHub (DVORA-AZARKOVICH/Narrative-Similarity).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Jerusalem College of TechnologyJerusalem · Israel13 works · 147 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Dvori Rosenfeld, Rinat Walles and Chaya Liebeskind. 2026. JCT at SemEval-2026 Task 4: A Multi-Method Approach to Narrative Story Similarity. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.387

@inproceedings{rosenfeld2026jct,
  title = {{JCT at SemEval-2026 Task 4: A Multi-Method Approach to Narrative Story Similarity}},
  author = {Dvori Rosenfeld, Rinat Walles and Chaya Liebeskind},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.387},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.387}
}

References

7 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anonymous. 2026. Enhancing structural mapping with llm-derived abstractions for analogical reasoning in narratives. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing. find
  2. David Bamman, Brendan O’Connor, and Noah A. Smith. 2013. Learning latent personas of film characters. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 352–361, Sofia, Bulgaria. Association for Computational Linguistics.
  3. Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150.
  4. Nathanael Chambers and Dan Jurafsky. 2008. Unsupervised learning of narrative event chains. In Proceedings of ACL-08: HLT, pages 789–797, Columbus, Ohio. Association for Computational Linguistics. find
  5. Dedre Gentner. 1983. Structure-mapping: A theoretical framework for analogy. Cognitive Science, 7(2):155– 170. find
  6. Hans Ole Hatzel, Ekaterina Artemova, Haimo Stiemer, Evelyn Gius, and Chris Biemann. 2026. SemEval-2025 task 4: Narrative similarity and narrative representation learning. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, CA, USA. find
  7. Hans Ole Hatzel and Chris Biemann. 2024. Story embeddings – narrative-focused representations of fictional stories. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5931–5943. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rosenfeld-2026-jct-semeval-2026-task-4/