Elamur Research⌕
☰
home / papers / 2026 / Takoyaki at SemEval-2026 Task 3: Ensembling LLM Predictions using Demonstration Retrieval for Dimensional Aspect-based Sentiment Analysis

Takoyaki at SemEval-2026 Task 3: Ensembling LLM Predictions using Demonstration Retrieval for Dimensional Aspect-based Sentiment Analysis

Paper · 2026 · Kosuke Yamada, Sho Takase and Ryosuke Kohita

Summary

The paper describes CyberAgent's "Takoyaki" system for SemEval-2026 Task 3 (DimABSA). The task is dimensional aspect-based sentiment analysis, where sentiment is expressed as continuous valence-arousal (VA) scores on a 1–9 scale rather than as categorical labels. The team entered Subtask 2 (DimASTE), which extracts aspect, opinion and VA triplets, and Subtask 3 (DimASQP), which adds an aspect category to form quadruplets. The data covers six languages (Chinese, English, Japanese, Russian, Tatar, Ukrainian) and restaurant, laptop and hotel reviews.

The system avoids fine-tuning entirely and relies on in-context learning with Gemini 3.0 Pro:

  • BM25 retrieves 50 similar training examples as demonstrations, under several tokenization variants.
  • An agreement-based ensemble merges the outputs of these variants.
  • For some datasets, LLM-mined correction rules refine predictions the ensemble is unsure about.

The system took the top position on three out of eight datasets in Subtask 3 and on two Subtask 2 datasets, beating the organizer baselines by a wide margin. A human study on Japanese hotel reviews found that Gemini agrees with gold labels more than humans do. It also found that exact-match cF1 understates practical output quality.

Key points

  • Retrieval variants: demonstrations are retrieved with character bigram, character trigram and word tokenization (jieba for Chinese, UniDic-lite for Japanese, whitespace for other languages). Japanese uses both English and Japanese prompts, giving n=6 variants; other languages use n=3.
  • Ensemble rule: a quadruplet is accepted if all n variants produce it and rejected if at most ⌊n/2⌋ do. Intermediate cases go to the optional correction step. VA scores are averaged across the agreeing variants.
  • Subtask 3 cF1 versus best baseline:
    • eng-rest: 65.14 vs 37.46 (rank 1)
    • ukr-rest: 50.19 vs 29.71 (rank 4/10)
    • jpn-hot: 40.86 (rank 2)
  • Subtask 2 cF1:
    • eng-rest: 70.21 (rank 1)
    • ukr-rest: 54.38 vs 42.20 (rank 5/12)
  • Other prompting strategies (jpn-hot): contrastive ICL raised cF1 only slightly (39.12 vs 37.51) and did not justify its cost. Cheat-sheet ICL lowered it to 36.49.
  • Human evaluation (80 jpn-hot instances, 3 annotators):
    • Exact F1 was 0.333 for Gold–Gemini, 0.240 for Gold–Human and 0.213 for Human–Human.
    • About 80.8% of the 169 Gemini triplets received quality scores of 4–5.
    • Krippendorff's α was 0.373.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSentiment & emotion · 972 worksInformation retrieval & search · 473 works

Cite

Cite as Kosuke Yamada, Sho Takase and Ryosuke Kohita. 2026. Takoyaki at SemEval-2026 Task 3: Ensembling LLM Predictions using Demonstration Retrieval for Dimensional Aspect-based Sentiment Analysis. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.219

@inproceedings{yamada2026takoyaki,
  title = {{Takoyaki at SemEval-2026 Task 3: Ensembling LLM Predictions using Demonstration Retrieval for Dimensional Aspect-based Sentiment Analysis}},
  author = {Kosuke Yamada, Sho Takase and Ryosuke Kohita},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.219},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.219}
}

References

32 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… find
  2. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient finetuning of quantized LLMs. In Advances in Neural Information Processing Systems, volume 36.
  3. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. 2024. A survey on in-context learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1107–1128.
  4. Gemini Team. 2023. Gemini: A family of highly capable multimodal models. ArXiv preprint arXiv:2312.11805.
  5. Ukyo Honda, Soichiro Murakami, and Peinan Zhang. 2025. Distilling many-shot in-context learning into a cheat sheet. In Findings of the Association for Computational Linguistics: EMNLP 2025. find
  6. Kimi Team. 2025. Kimi K2: Open agentic intelligence. ArXiv preprint arXiv:2507.20534.
  7. Klaus Krippendorff. 2011. Computing Krippendorff’s alpha-reliability. Departmental Papers (ASC). find
  8. Lung-Hao Lee, Liang-Chih Yu, Natalia Loukashevich, Ilseyar Alimova, Alexander Panchenko, Tzu-Mi Lin, Zhe-Yu Xu, Jian-Yu Zhou, Guangmin Zheng, Jin Wang, Sharanya Awasthi, Jonas Becker, Jan Philip Wahle, Terry Ruas, Shamsuddeen Hassan Muhammad, and Saif M. Mohammad. 2026. DimABSA: Building multilingua… find
  9. Paul McCann. 2020. fugashi, a tool for tokenizing Japanese in python. In Proceedings of Second Workshop for NLP Open Source Software, pages 44–51, Online. Association for Computational Linguistics. find
  10. Ling-ang Meng, Tianyu Zhao, and Dawei Song. 2024. DS-group at SIGHAN-2024 dimABSA task: Constructing in-context learning structure for dimensional aspect-based sentiment analysis. In Proceedings of the 10th SIGHAN Workshop on Chinese Language Processing, pages 127–132. find
  11. Ying Mo, Jiahao Liu, Jian Yang, Qifan Wang, Shun Zhang, Jingang Wang, and Zhoujun Li. 2024. C-ICL: Contrastive in-context learning for information extraction. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10099– 10114. find
  12. OpenAI. 2025. OpenAI GPT-5 system card. ArXiv preprint arXiv:2601.03267.
  13. Haiyun Peng, Lu Xu, Lidong Bing, Fei Huang, Wei Lu, and Luo Si. 2020. Knowing what, how and why: A near complete solution for aspect-based sentiment analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8600–8607. find
  14. Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar, Mohammad AL-Smadi, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphée De Clercq, Véronique Hoste, Marianna Apidianaki, Xavier Tannier, Natalia Loukachevitch, Evgeniy Kotelnikov, Nuria Bel, Salud María Jiménez-Z… — SemEval-2016 Task 5: Aspect Based Sentiment Analysis
  15. Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Suresh Manandhar, and Ion Androutsopoulos. 2015. SemEval-2015 task 12: Aspect based sentiment analysis. In Proceedings of the 9th International Workshop on Semantic Evaluation, pages 486–495.
  16. Lung-Hao Lee, Liang-Chih Yu, Suge Wang, and Jian Liao. 2024. Overview of the SIGHAN 2024 shared task for Chinese dimensional aspect-based sentiment analysis. In Proceedings of the 10th SIGHAN Workshop on Chinese Language Processing, pages 165– 174. find
  17. Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. SemEval-2014 task 4: Aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation, pages 27–35.
  18. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out, pages 100– 114.
  19. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333–389. find
  20. Man Luo, Xin Xu, Zhuyun Dai, Panupong Pasupat, Mehran Kazemi, Chitta Baral, Vaiva Imbrasaite, James A. Russell. 1980. A circumplex model of affect. Journal of Personality and Social Psychology, 39(6):1161–1178. find
  21. James A. Russell. 2003. Core affect and the psychological construction of emotion. Psychological Review, 110(1):145–172. find
  22. Jakub Šmíd, Pavel Priban, and Pavel Kral. 2024. LLaMA-based models for aspect-based sentiment analysis. In Proceedings of the 14th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, pages 63–70. find
  23. Junyi Sun. 2012. jieba: Chinese text segmentation tool. https://github.com/fxsjy/jieba. ↗
  24. Hongling Xu, Delong Zhang, Yice Zhang, and Ruifeng Xu. 2024. HITSZ-HLT at SIGHAN-2024 dimABSA task: Integrating BERT and LLM for Chinese dimensional aspect-based sentiment analysis. In Proceedings of the 10th SIGHAN Workshop on Chinese Language Processing, pages 175–185. find
  25. Lu Xu, Hao Li, Wei Lu, and Lidong Bing. 2020. Position-aware tagging for aspect sentiment triplet extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2339–2349.
  26. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. ArXiv preprint arXiv:2505.09388.
  27. Liang-Chih Yu, Jonas Becker, Shamsuddeen Hassan Muhammad, Idris Abdulmumin, Lung-Hao Lee, Ying-Lung Lin, Jin Wang, Jan Philip Wahle, Terry Ruas, Alexander Panchenko, Ilseyar Alimova, Kai-Wei Chang, Lilian Wanzare, Nelson Odhiambo, Bela Gipp, and Saif M. Mohammad. 2026. SemEval-2026 task 3: Dimension… find
  28. Wenxuan Zhang, Yang Deng, Xin Li, Yifei Yuan, Lidong Bing, and Wai Lam. 2021. Aspect sentiment quad prediction as paraphrase generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9209– 9219.
  29. Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Pan, and Lidong Bing. 2024. Sentiment analysis in the era of large language models: A reality check. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3881–3906, Mexico City, Mexico. Association for Computational Linguistics.
  30. Wenxuan Zhang, Xin Li, Yang Deng, Lidong Bing, and Wai Lam. 2023. A survey on aspect-based sentiment analysis: Tasks, methods, and challenges. IEEE Transactions on Knowledge and Data Engineering, 35(11):11019–11038. find
  31. Changzhi Zhou, Dandan Song, Yuhang Tian, Zhijing Wu, Hao Wang, Xinyu Zhang, Jun Yang, Ziyi Yang, and Shuhao Zhang. 2024. A comprehensive evaluation of large language models on aspect-based sentiment analysis. ArXiv preprint arXiv:2412.02279. — [arXiv:2412.02279](https://arxiv.org/abs/2412.02279) find
  32. Senbin Zhu, Hanjie Zhao, Xingren Wang, Shanhong Liu, Yuxiang Jia, and Hongying Zan. 2024. ZZU-NLP at SIGHAN-2024 dimABSA task: Aspect-based sentiment analysis with coarse-to-fine in-context learning. In Proceedings of the 10th SIGHAN Workshop on Chinese Language Processing, pages 112–120. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yamada-2026-takoyaki-semeval-2026-task-3/