HERMeS: Human Evaluation & Ranking of MultiplE Systems

Summary

Comparing many MT engines and LLMs through human judgment is laborious. Annotators tire, their ratings drift, and existing tools are not always built for side-by-side scoring of a dozen or more anonymized outputs. HERMeS is a lightweight, open-source web tool for this scenario.

Each source sentence appears on a card with its candidate translations. Evaluators work in two steps. First, they sort every translation into one of four coarse quality buckets. Second, they give it a numeric direct-assessment score within that bucket's range. The tool can automatically group and reorder items as judgments are made, and evaluators can add notes per sentence.

Inputs are randomized and anonymized, and hash checks expose tampering and make it safe to merge annotations collected separately. The app keeps no state and requires no login, so evaluators save their progress as downloadable spreadsheet checkpoints.

The author positions HERMeS as a complement to Appraise, not a replacement, and it does not support detailed error categorization. Early qualitative feedback suggests the bucket-first approach may lower perceived effort and improve consistency. No quantitative results are reported.

Key points

Two-stage workflow: coarse ranking into four buckets (Best, Good, OK, Poor; names configurable), then direct assessment on a configurable scale. In the author's study, each bucket covers 7 points (e.g., 1–7 for Poor, 8–14 for OK).
Data integrity: sources are randomized, anonymized and hash-verified before evaluation, and the research team de-anonymizes and de-randomizes the returned results.
Infrastructure: built with Streamlit, hosted on Streamlit Community Cloud, free to use, with no login and no server-side storage. Work is saved and resumed via spreadsheet checkpoints, and the code is on GitHub.
Pilot study: 13 systems × 100 sentences = 1,300 translations, with only preliminary qualitative feedback reported.
Positioning: meant to complement Appraise (Federmann, 2012) for large multi-system comparison, not for fine-grained error analysis.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation · Evaluation & benchmarks

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{vanhorn2026hermes,
  title = {{HERMeS: Human Evaluation \& Ranking of MultiplE Systems}},
  author = {Vanhorn, Rex},
  booktitle = {Proceedings of the 26th Annual Conference of the European Association for Machine Translation (Volume 2)},
  year = {2026},
  url = {https://aclanthology.org/2026.eamt-2.8/}
}

References

2 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Castilho, S. (2021, April). Towards document-level human MT evaluation: On the issues of annotator agreement, effort and misevaluation. In Proceedings of the workshop on human evaluation of NLP systems (HumEval) (pp. 34– 45). find
Federmann, C. (2012). Appraise: an Open-Source Toolkit for Manual Evaluation of MT Output. Prague Bull. Math. Linguistics, 98, 25–36. ——————————————————————— 2 https://streamlit.io 3 https://mt-eval.streamlit.app/?type=admin 4 https://github.com/RexVH/St-MT-Evaluation-app ↗

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vanhorn-2026-hermes-human-evaluation-ranking-multiple/