Elamur Research⌕
☰
home / papers / 2026 / CiNet-Handai-Kyodai at SemEval-2026 Task 5: Combining LLM Prompting, Semantic Similarity, and Synthetic Gaze for Graded Sense Plausibility

CiNet-Handai-Kyodai at SemEval-2026 Task 5: Combining LLM Prompting, Semantic Similarity, and Synthetic Gaze for Graded Sense Plausibility

Paper · 2026 · Lis Kanashiro Pereira and Fei Cheng

Summary

This system description addresses SemEval-2026 Task 5, which asks how plausible a given word sense is for a homonym inside a short English story. Several senses may fit at once, so systems output a 1–5 rating rather than a single label. The task uses the AmbiStory dataset. The authors observe that direct LLM prompting tends to favour dominant senses and miss subtle narrative cues.

Their system starts with a GPT-4.1 zero-shot plausibility rating and adds three auxiliary signals:

  • embedding-based similarity features from OpenAI text-embedding-3-large;
  • a story-specific dictionary-style definition generated by the LLM and compared with the candidate sense;
  • a synthetic gaze value, namely predicted total fixation time on the target word.

An ordinary least squares regressor trained on the training split combines these signals. The best configuration reaches 90.10 Acc±SD and 79.19 Spearman on the test set. Its Acc±SD exceeds the human reference, while its rank correlation stays below it. The work shows that cheap semantic and cognitively motivated features can improve LLM judgments on graded lexical-semantic tasks.

Key points

  • AmbiStory has 3,798 instances (633 setups of six), split 2,280 / 588 / 930 for train / dev / test. Splits are stratified by homonym to prevent lexical overlap.
  • On test, scores rise as features are added:
    • Prompt + EmbSim: 89.46 / 76.09
    • adding Def.Gen.: 89.89 / 78.87
    • adding TFT: 90.10 / 79.19
    • human reference: 89.20 / 83.40
  • Synthetic gaze comes from the RoBERTa-based model of Li and Rudzicz (2021). Only total fixation time (TFT) on the target word is used.
  • Fine-tuned baselines performed much worse on dev: DeBERTa-v3-large scored 66.03 / 66.03 and gpt-oss-20b scored 73.00 / 65.00.
  • The TFT analysis shows where gaze helps and where it hurts:
    • It lowers over-prediction bias (mean error 0.06 → 0.02).
    • It improves low-plausibility items.
    • It hurts highly plausible idiomatic cases, such as "got to the point".

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Kyoto UniversityKyoto · Japan40 works · 8,235 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Lis Kanashiro Pereira and Fei Cheng. 2026. CiNet-Handai-Kyodai at SemEval-2026 Task 5: Combining LLM Prompting, Semantic Similarity, and Synthetic Gaze for Graded Sense Plausibility. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.245

@inproceedings{pereira2026cinet,
  title = {{CiNet-Handai-Kyodai at SemEval-2026 Task 5: Combining LLM Prompting, Semantic Similarity, and Synthetic Gaze for Graded Sense Plausibility}},
  author = {Lis Kanashiro Pereira and Fei Cheng},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.245},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.245}
}

References

4 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. \# CiNet-Handai-Kyodai at SemEval-2026 Task 5: Combining LLM Prompting, Semantic Similarity, and Synthetic Gaze for Graded Sense Plausibility Lis Kanashiro Pereira, Fei Cheng · Proceedings of the 20th International Workshop on Semantic Evaluation (2026) · 2026 find
  2. Source: https://aclanthology.org/2026.semeval-1.245/ · License: CC BY 4.0 (ACL Anthology). Text extracted automatically from the PDF (poppler): reading-order and hyphenation errors are possible; formulas are Unicode, not LaTeX. CiNet-Handai-Kyodai at SemEval-2026 Task 5: Combining LLM Prompting, Sem… — [ACL 2026.semeval-1.245](https://aclanthology.org/2026.semeval-1.245/) ↗
  3. Task Description SemEval-2026 Task 5 (Gehring et al., 2026) focuses on rating the plausibility of word senses in lexically ambiguous sentences through narrative understanding. Unlike traditional Word Sense Disambiguation (WSD), which typically assumes a single *Figure 1: Overview of SemEval-2026 Tas… find
  4. Method Our system is a hybrid of prompt-based plausibility rating and auxiliary features derived from embeddings and synthetic gaze. Specifically, we: (i) query an LLM with a prompt based on the AmbiStory prompt format (Gehring and Roth, 2025) to obtain a plausibility score on the 1–5 scale (§3.1); find

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pereira-2026-cinet-handai-kyodai-semeval-2026/