Elamur Research⌕
☰
home / papers / 2026 / Evaluating Prompt Scope and Demonstration Similarity in Local LLM Machine Translation

Evaluating Prompt Scope and Demonstration Similarity in Local LLM Machine Translation

Paper · 2026 · Mihael Arcan

Summary

LLMs are often used as translators, yet they are usually tested with one prompt shape: one English sentence in, one target language out. This paper asks whether the scope of the request and the choice of in-context examples should count as evaluation variables in their own right, focusing on small local models. Three Ollama-served LLMs (llama3.2:3b, mistral:latest, qwen2.5:14b) translate the full FLORES-200 devtest from English into five Romance and four Germanic EU languages. They are compared against OPUS-MT/MarianMT and NLLB-200 baselines. Prompts are either single-target or family-scope, meaning all languages of a family returned together as JSON. Each prompt is zero-shot or uses five demonstrations, chosen at random, by lexical similarity or by embedding similarity. Dedicated MT systems still win overall, most clearly on Germanic. Few-shot examples help the two larger LLMs but harm the 3B model. Embedding retrieval gives only a small edge. Family-scope prompting works for stronger models, but smaller ones often skip languages. The study argues that reports on LLM translation should cover prompt scope and output compliance, not only quality scores.

Key points

  • Data: FLORES-200, with demonstrations taken only from dev and evaluation on all 1012 devtest sentences. A source audit checks for leakage. The Germanic targets are Swedish, Danish, Dutch and German, and the Romance targets are Romanian, Portuguese, Italian, Spanish and French.
  • Systems: llama3.2:3b, mistral:latest and qwen2.5:14b, run at temperature 0. The baselines, nllb-200-distilled-600M, nllb-200-distilled-1.3B and OPUS-MT, are run in single-target mode only.
  • Metrics: SacreBLEU BLEU, chrF++ and COMET (Unbabel/wmt22-comet-da). Paired bootstrap gives 95% CIs for the best LLM against the best MT baseline. Family prompts also get coverage, complete-output, missing-target and prompt-echo counts.
  • Retrieval: embedding-based example selection gives the best mean gains for mistral and qwen. However, the per-sentence correlation between similarity score and quality gain is close to zero.
  • Failure case: llama3.2:3b translates acceptably one language at a time, but frequently omits languages when asked for the whole Germanic family in one response.

Interesting ideas

  • I1. Random demonstrations serve as a required control, which separates the benefit of relevant examples from the benefit of simply showing the task pattern.
  • I2. The paper checks whether a per-item similarity score predicts per-item improvement, instead of trusting averages alone; a positive mean gain can hide a near-zero link.
  • I3. Multi-output requests are scored on two axes, quality of what was produced and completeness (missing targets, echoes), because quality computed only over returned items can flatter a model.
  • I4. Task files are deterministic and carry every condition's metadata, and each evaluation stage (quality, compliance, significance, retrieval analysis) can be re-run independently.
  • I5. Asking for several related target languages at once tests whether a model can coordinate outputs under one schema, which exposes cross-target interference.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When LLM teacher tiers are given retrieved examples, compare them against randomly chosen examples so that any gain credited to retrieval has actually been earned.
  • I2 (learn): Correlate the confidence of silver-data agreement or repair-rule triggers with per-sentence LAS change, to test whether a signal truly explains the gains.
  • I3 (back): When generating several renderings from one meaning graph, record coverage and echo of the source graph alongside fluency, so that partial outputs are not counted as successes.
  • I4 (learn): Store parser and teacher runs as JSONL with prompt, scope and example metadata, so that each evaluation step can be regenerated and audited apart from the others.
  • I5 (back): Translation through the meta-world could be checked for consistency by producing related languages from one world state and flagging targets that diverge or go missing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Ollscoil na Gaillimhe – University of GalwayGalway · Ireland23 works · 2,229 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Machine translation · 1,087 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Mihael Arcan. 2026. Evaluating Prompt Scope and Demonstration Similarity in Local LLM Machine Translation. arXiv:2607.26286.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2607.26286 · arXiv 2607.26286

@misc{arcan2026evaluating,
  title = {{Evaluating Prompt Scope and Demonstration Similarity in Local LLM Machine Translation}},
  author = {Mihael Arcan},
  year = {2026},
  doi = {10.48550/arxiv.2607.26286},
  eprint = {2607.26286},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2607.26286}
}

References

19 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mihael Arcan, Daniel Torregrosa, Sina Ahmadi, and John P. McCrae. Inferring translation candidates for multilingual dictionary generation with multi-way neural machine translation. In Proceedings of the Translation Inference Across Dictionaries 2019 Shared Task, volume 2493 of CEUR Workshop Proceedings, pages 13–23, 2019. find
  2. Mihael Arcan, Daniel Torregrosa, and Paul Buitelaar. Translating terminological expressions in knowledge bases with neural machine translation. arXiv preprint arXiv:1709.02184, 2017. ↗
  3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901, 2020.
  4. Bharathi Raja Chakravarthi, Mihael Arcan, and John P. McCrae. WordNet gloss translation for under-resourced languages using multilingual neural machine translation. In Proceedings of the Second Workshop on Multilingualism at the Intersection of Knowledge Bases and Machine Translation, pages 1–7, Dublin, Ireland, 2019. European Association for Machine Translation. find
  5. Ali Hatami, Mihael Arcan, and Paul Buitelaar. Enhancing translation quality by leveraging semantic diversity in multimodal machine translation. In Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pages 154–166, Chicago, USA, 2024. Association for Machine Translation in the Americas. find
  6. Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. How good are GPT models at machine translation? a comprehensive evaluation. In arXiv preprint arXiv:2302.09210, 2023. ↗
  7. Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, Andre F. T. Martins, and Alexandra Birch. Marian: Fast neural machine translation in C++. In Proceedings of ACL 2018, System Demonstrations, pages 116–121. Association for Computational Linguistics, 2018.
  8. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out, pages 100–114. Association for Computational Linguistics, 2022. ↗
  9. John P. McCrae, Mihael Arcan, and Paul Buitelaar. Linking knowledge graphs across languages with semantic similarity and machine translation. In Proceedings of the First Workshop on Multi-Language Processing in a Globalising World, 2017. find
  10. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, et al. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672, 2022. ↗
  11. Maja Popovic. chrF: Character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal, 2015. Association for Computational Linguistics.
  12. Maja Popovic. chrF++: Words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, pages 612–618, Copenhagen, Denmark, 2017. Association for Computational Linguistics.
  13. Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Brussels, Belgium, 2018. Association for Computational Linguistics.
  14. Ricardo Rei, Jose G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C. Farinha, Taisiya Glushkova, Alon Lavie, and Luisa Coheur. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation, pages 578–585. Association for Computational Linguistics, 2022.
  15. Ricardo Rei, Craig Stewart, Ana C. Farinha, and Alon Lavie. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2685–2702. Association for Computational Linguistics, 2020.
  16. Jörg Tiedemann and Santhosh Thottingal. The OPUS-MT project: Building open translation services for the world. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 479–480, 2020. find
  17. Daniel Torregrosa, Nivranshu Pasricha, Maraim Masoud, Bharathi Raja Chakravarthi, Juan Alonso, Noe Casas, and Mihael Arcan. Leveraging rule-based machine translation knowledge for under-resourced neural machine translation models. In Proceedings of Machine Translation Summit XVII: Translator, Project and User Tracks, pages 125–133, Dublin, Ireland, 2019. European Association for Machine Translation.
  18. Daniel Torregrosa, Nivranshu Pasricha, Maraim Masoud, Bharathi Raja Chakravarthi, Juan Alonso, Noe Casas, and Mihael Arcan. Aspects of terminological and named entity knowledge within rule-based machine translation models for under-resourced neural machine translation scenarios. arXiv preprint arXiv:2009.13398, 2020. ↗
  19. Biao Zhang, Barry Haddow, and Alexandra Birch. Prompting large language model for machine translation: A case study. arXiv preprint arXiv:2301.07069, 2023. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/arcan-2026-evaluating-prompt-scope-demonstration-similarity/