Elamur Research⌕
☰
home / papers / 2026 / High-Order Question Generation in a Multilingual Educational Context

High-Order Question Generation in a Multilingual Educational Context

Summary

Teachers rarely ask questions that demand analysis, evaluation or creation, partly because such questions take effort to write. This study tests whether LLMs can draft them, and it extends earlier work beyond English and beyond Bloom's Taxonomy. The authors used 30 secondary-school physics texts (10 each in Basque, Spanish and English) from the Agrega2 repository, aimed at pupils aged 12–16 in the Basque Autonomous Community. Llama3.1_70B and GPT-4 received chain-of-thought prompts grounded in three frameworks: Bloom's Taxonomy, Claim-Evidence-Reasoning (CER) and divergent questioning. Secondary-school teachers rated the 300 resulting questions. The authors then examined the form and content of the questions teachers judged high-order. Most questions could be answered, but teachers saw only about half of the answerable ones as genuinely high-order. Prompts based on Bloom's "Evaluate" level and on divergent questioning worked best for higher-order output. CER and divergent prompts yielded questions that differed in structure and concept from Bloom-based ones. This suggests the frameworks can complement each other, and it shows that human review of LLM-generated questions remains necessary.

Key points

  • Data: 30 physics texts from the Agrega2 project, which is EU-cofunded and was previously used for multilingual readability research. Each model produced five questions per text, giving 300 questions in total.
  • Teacher evaluation: Four teachers took part. Three shared the full set, and a fourth re-rated 90 questions (30 per language) to measure agreement. The four metrics were Answerability, Relevance, Bloom's Taxonomy Level and Teacher Adoption.
  • Author analysis: Questions labelled high-order were coded on six criteria: task type, single vs. nested questions, lexical and structural patterns, alignment with prompt wording, explicit reference to the source text, and length.
  • Model and language results: Each model was able to produce questions in Basque, Spanish and English alike. GPT-4 was slightly ahead in Spanish and English, while Llama3 did slightly better in Basque.
  • Limitations:
    • Cohen's Kappa was low despite moderate-to-high raw agreement. Gwet's AC1/AC2 indicated a mild kappa paradox caused by skewed ratings.
    • Using Bloom levels as the rating scale may favour the Bloom-based prompts.
    • The study covers only one domain (physics) and two models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/56rihjwb6jq7

@inproceedings{ucar2026high,
  title = {{High-Order Question Generation in a Multilingual Educational Context}},
  author = {Uçar, Suna Seyma and Aldabe, Itziar and Aranberri, Nora and Clercq, Orphee De},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/56rihjwb6jq7},
  url = {https://doi.org/10.63317/56rihjwb6jq7}
}

References

16 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Donna Rae Clasen and Robert E. clasen. 1990. Teachers trackle thinking. find
  2. Zulmaris Diaz, Michael Whitacre, J Joy Esquierdo, and Jose A Ruiz-Escalante. 2013. Why did i ask that question? bilingual/esl pre-service teachers’ insights. International Journal of instruction, 6(2). find
  3. Sabina Elkins, Ekaterina Kochmar, Jackie CK Cheung, and Iulian Serban. 2024. How teachers can use large language models and bloom’s taxonomy to create educational quizzes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 23084–23091. find
  4. Sabina Elkins, Ekaterina Kochmar, Iulian Serban, and Jackie CK Cheung. 2023. How useful are educational questions generated by large language models? In International Conference on Artificial Intelligence in Education, pages 536–542. Springer. find
  5. Alvan R Feinstein and Domenic V Cicchetti. 1990. High agreement but low kappa: I. the problems of two paradoxes. Journal of clinical epidemiology, 43(6):543–549. find
  6. Meredith D Gall. 1970. The use of questions in teaching. Review of educational research, 40(5):707–721. find
  7. James J Gallagher and Mary Jane Aschner. 1963. A preliminary report on analyses of classroom interaction. Merrill-Palmer Quarterly of Behavior and Development, 9(3):183–194. find
  8. Kilem Li Gwet. 2008. Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1):29–48. find
  9. DR Krathwohl. 2002. A revision bloom’s taxonomy: An overview. Theory into Practice. find
  10. Joanne Gainen Kurfiss. 1988. Critical thinking: Theory, research, practice, and possibilities. ashe-eric higher education report no. 2, 1988. find
  11. David R McNamara. 1981. Teaching skill: the question of questi. Educational Research, 23(2):104– 109. find
  12. Katherine L McNeill, David J Lizotte, Joseph Krajcik, and Ronald W Marx. 2006. Supporting students’ construction of scientific explanations by fading scaffolds in instructional materials. The journal of the Learning Sciences, 15(2):153–191. find
  13. Robert T Pate and Neville H Bremer. 1967. Guiding learning through skilful questioning. The Elementary School Journal, 67(8):417–422. find
  14. Luise B Savage. 1998. Eliciting critical thinking skills through questioning. The Clearing House, 71(5):291–293. find
  15. Nicy Scaria, Suma Dharani Chenna, and Deepak Subramani. 2024. Automated educational question generation at different bloom’s skill levels using large language models: Strategies and evaluation. In International Conference on Artificial Intelligence in Education, pages 165–179. Springer. find
  16. Suna-Şeyma Uçar, Itziar Aldabe, Nora Aranberri, and Ana Arruarte. 2024. Exploring automatic readability assessment for science documents within a multilingual educational context. International Journal of Artificial Intelligence in Education, pages 1–43. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ucar-2026-high-order-question-generation-multilingual/