Elamur Research⌕
☰
home / papers / 2026 / AMSN at PFB: Overview of the Automated Multilingual Economics Question Answering Task

AMSN at PFB: Overview of the Automated Multilingual Economics Question Answering Task

Paper · 2026 · Ali Saleh Mohammadabad and Mehregan Nazarmohsenifakori

Summary

This EVALITA 2026 report describes one participating team's system for the AMSN shared task, despite "Overview" in its title. The task is multiple-choice economics question answering in English, Italian and Turkish. Each item has five options (A–E, where E is usually a "none of the above" choice). Topics span micro- and macroeconomics, finance and quantitative methods, with 1,001 test questions per language.

The authors compare three strategies:

  • fine-tuning mDeBERTa-v3;
  • querying GPT-4o and GPT-5 through the OpenAI API with a chain-of-thought system prompt;
  • a hybrid in which transformer classifiers flag likely errors in GPT-4o's output and substitute a new answer.

Fine-tuning plateaus near 50%. GPT-4o with chain-of-thought gets 74% on English and Italian validation, and the hybrid raises this to roughly 78%. GPT-5 alone beats all of these, and the correction layer adds nothing on top of it. Zero-shot GPT-5 was therefore submitted, averaging 89.54% on test. The main takeaway is that a stronger foundation model can make engineered post-correction unnecessary. However, the results depend on a proprietary model whose exact snapshot is not reported.

Key points

  • Prompt selection: Five prompting strategies were compared on only 50 validation questions. Chain of Thought and Expert Reasoning tied at 90%. Final settings were temperature 0.1 and up to 1500 output tokens, with regex-style extraction of an "ANSWER: X" line.

  • Validation accuracy: GPT-4o scored 74/74/65% and GPT-5 scored 82/81/73% (EN/IT/TR). Turkish trails the other two languages by roughly 8–9 points for both models.

  • Test accuracy (GPT-5):

    • EN: 89.11% (892/1,001)
    • IT: 91.21% (913/1,001)
    • TR: 88.31% (884/1,001)

    These are 7–15 points above validation. The authors attribute the gap to differences between the data splits rather than leakage, but they offer no verification.

  • Hybrid system: It consists of a binary mistake detector plus a five-way answer corrector, both trained on GPT-4o's validation predictions. The reported gain is inconsistent: 3–5 pp in the results section but about 6.5 pp in the conclusion. No Turkish figure is given, so its 77.5% average covers only EN and IT.

  • Caveats:

    • The GPT-5 version string is left as a placeholder, which hinders reproducibility.
    • Option E is predicted in only 1–2% of cases, while options A–C account for about 82–83% of predictions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 3Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksMultilingual & cross-lingual NLP · 828 worksQuestion answering · 441 works

Cite

Cite as Ali Saleh Mohammadabad and Mehregan Nazarmohsenifakori. 2026. AMSN at PFB: Overview of the Automated Multilingual Economics Question Answering Task. Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{mohammadabad2026amsn,
  title = {{AMSN at PFB: Overview of the Automated Multilingual Economics Question Answering Task}},
  author = {Ali Saleh Mohammadabad and Mehregan Nazarmohsenifakori},
  booktitle = {Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026)},
  year = {2026},
  url = {https://aclanthology.org/2026.evalita-1.60/}
}

References

3 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. P. He, J. Gao, W. Chen, Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing, in: International Conference on Learning Representations (ICLR), 2023. find
  2. OpenAI, Gpt-4 technical report, arXiv preprint arXiv:2303.08774 (2023).
  3. J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, D. Zhou, Chain-of-thought prompting elicits reasoning in large language models, in: Advances in Neural Information Processing Systems (NeurIPS), volume 35, 2022, pp. 24824–24837.

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mohammadabad-2026-amsn-pfb-overview-automated-multilingual/