Elamur Research⌕
☰
home / papers / 2026 / RETUYT-INCO at BEA 2026 Shared Task 2: Meta-prompting in Rubric-based Scoring for German

RETUYT-INCO at BEA 2026 Shared Task 2: Meta-prompting in Rubric-based Scoring for German

Paper · 2026 · Ignacio Sastre, Ignacio Remersaro, Facundo Díaz et al.

Summary

The authors (RETUYT-INCO) took part in the BEA 2026 shared task on rubric-based scoring of short German student answers in science and mathematics. They competed in three tracks: Track 1, which scores new answers to already-seen questions on a three-way scale, and Tracks 3 and 4, which both use a binary scale, the former for new answers to known questions and the latter for answers to questions never seen in training. A generic Gemini 3 Flash scoring prompt performed unevenly across the 72 question-rubric groups in the training data. To address this, they built a "meta-prompting" pipeline. In an offline step, an LLM reads a group's question, its rubric and its labelled training answers, then writes a scoring prompt tailored to that group. At inference, each new answer goes to its group's prompt. For every group, they kept the best of several candidate prompts, which differed in wording, thinking budget and whether synthetic answers were added. They also compared SVM with TF-IDF, LoRA fine-tuning and prompt tuning of Llama 3.1 8B, and a multi-call role-playing prompt. Meta-prompting was their strongest unseen-answers method, and the systems finished mid-table. The work shows that per-group induced prompts can beat a single static prompt. It also shows a wide gap between proprietary and small open models.

Key points

  • Meta-prompting on the Trial set: choosing the best variant per group raised QWK from 0.563 with the generic Gemini 3 Flash baseline to 0.743. Weighted F1 rose from 0.820 to 0.892.
  • Official results:
    • Track 1: 6th of 8 teams, QWK 0.729.
    • Track 3: 4th of 9 teams, QWK 0.674.
    • Track 4: 4th of 8 teams, QWK 0.490, achieved by the plain Gemini baseline.
  • Synthetic data: generated with LangGraph/LangChain, using student role-play, noise injection and rubric-based validation. It was applied to the 25 most imbalanced questions to halve the class gap.
  • Llama 3.1 8B: plain prompting scored QWK 0.050 on Trial. Prompt tuning with one learned embedding per group reached 0.583 and beat a global LoRA run. However, it dropped on Test, likely from overfitting to the Trial set used for early stopping.
  • Limitations: the methods depend on closed APIs, which raises cost, privacy and access concerns, especially for minors and rural schools. The three-way setting was explored less than the two-way tracks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Universidad de la República de UruguayMontevideo · Uruguay9 works · 450 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksEducation · 203 works

Cite

Cite as Ignacio Sastre, Ignacio Remersaro, Facundo Díaz, Nicolás De Horta, Luis Chiruzzo, Aiala Rosá and Santiago Góngora. 2026. RETUYT-INCO at BEA 2026 Shared Task 2: Meta-prompting in Rubric-based Scoring for German. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bea-1.91

@inproceedings{sastre2026retuyt,
  title = {{RETUYT-INCO at BEA 2026 Shared Task 2: Meta-prompting in Rubric-based Scoring for German}},
  author = {Ignacio Sastre, Ignacio Remersaro, Facundo Díaz, Nicolás {De Horta}, Luis Chiruzzo, Aiala Rosá and Santiago Góngora},
  booktitle = {Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.bea-1.91},
  url = {https://doi.org/10.18653/v1/2026.bea-1.91}
}

References

17 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alex Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. 2026. GEPA: Reflective prompt evolution can outperform reinforcement learning. In The Fourteenth International Conference on Learning Representations. ↗
  2. Xiaoyu Bai and Manfred Stede. 2023. A Survey of Current Machine Learning Approaches to Student Free-Text Evaluation for Intelligent Tutoring. International Journal of Artificial Intelligence in Education, 33(4):992–1030. ↗
  3. Alexis Baladón, Ignacio Sastre, Luis Chiruzzo, and Aiala Rosá. 2023. RETUYT-InCo at BEA 2023 shared task: Tuning open-source LLMs for generating teacher responses. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 756–765, Toronto, Canada. Association for Computational Linguistics. find
  4. Steven Burrows, Iryna Gurevych, and Benno Stein. 2015. The Eras and Trends of Automatic Short Answer Grading. International Journal of Artificial Intelligence in Education, 25(1):60–117. ↗
  5. Luis Chiruzzo, Laura Musto, Santiago Gongora, Brian Carpenter, Juan Filevich, and Aiala Rosa. 2022. Using NLP to support English teaching in rural schools. In Proceedings of the Second Workshop on NLP for Positive Impact (NLP4PI), pages 113–121, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  6. Myroslava Dzikovska, Rodney Nielsen, Chris Brew, Claudia Leacock, Danilo Giampiccolo, Luisa Bentivogli, Peter Clark, Ido Dagan, and Hoa Trang Dang. 2013. SemEval-2013 task 7: The joint student response analysis and 8th recognizing textual entailment challenge. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 263–274, Atlanta, Georgia, USA. Association for Computational Linguistics.
  7. Sebastian Gombert, Zhifan Sun, Fabian Zehner, Jannik Lossjew, Tobias Wyrwich, Berrit Katharina Czinczel, David Bednorz, Sascha Bernholt, Knut Neumann, Ute Harms, Aiso Heinze, and Hendrik Drachsler. 2026. Report on the bea 2026 shared task on rubric-based short answer scoring for german. In Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026). Association for Computational Linguistics.
  8. Santiago Góngora, Ignacio Sastre, Santiago Robaina, Ignacio Remersaro, Luis Chiruzzo, and Aiala Rosá. 2025. RETUYT-INCO at BEA 2025 shared task: How far can lightweight models go in AI-powered tutor evaluation? In Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2025), pages 1135–1144, Vienna, Austria. Association for Computational Linguistics. ↗
  9. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. ↗
  10. Or Honovich, Uri Shaham, Samuel R. Bowman, and Omer Levy. 2022. Instruction induction: From few examples to natural language task descriptions. Preprint, arXiv:2205.10782. ↗
  11. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  12. Aiala Rosá, Santiago Góngora, Juan Pablo Filevich, Ignacio Sastre, Laura Musto, Brian Carpenter, and Luis Chiruzzo. 2025. A platform for generating educational activities to teach english as a second language. Preprint, arXiv:2504.20251. ↗
  13. Ignacio Sastre, Leandro Alfonso, Facundo Fleitas, Federico Gil, Andrés Lucas, Tomás Spoturno, Santiago Góngora, Aiala Rosá, and Luis Chiruzzo. 2024. RETUYT-INCO at MLSP 2024: Experiments on language simplification using embeddings, classifiers and large language models. In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), pages 618– 626, Mexico City, Mexico. Association for Computational Linguistics. find
  14. Ignacio Sastre and Aiala Rosá. 2026. Concept Tokens: Learning behavioral embeddings through concept definitions. In Findings of the Association for Computational Linguistics: ACL 2026. Association for Computational Linguistics. ↗
  15. Yi Wang, Fenghua Weng, Sibei Yang, Zhan Qin, Minlie Huang, and Wenjie Wang. 2025a. DELMAN: Dynamic defense against large language model jailbreaking with model editing. In Findings of the Association for Computational Linguistics: ACL 2025, pages 11465–11481, Vienna, Austria. Association for Computational Linguistics. find
  16. Zhenhua Wang, Wei Xie, Shuoyoucheng Ma, Enze Wang, and Baosheng Wang. 2025b. Evading llms’ safety boundary with adaptive role-play jailbreaking. Electronics, 14(24). ↗
  17. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sastre-2026-retuyt-inco-bea-2026-shared/