基于事实验证的五链式机器翻译幻觉检测智能体

Summary

Machine translation hallucinations are outputs containing content unrelated to the source sentence. They are hard to catch with current tools. Internal-state detectors (Seq-Logprob, ALTI+, Attn-OT) require attention-based "white-box" models. External detectors (quality estimation, embedding similarity, NLI) are optimized for other objectives and overlook factual mismatch. Wei and Li propose MTHaluAgent, a LangChain-based LLM agent that uses only the source and its translation, so it also works with black-box systems such as GPT.

A screening chain queries the LLM four times, and a unanimous verdict is final. If the votes disagree, the agent extracts atomic facts from the translation and discards "invalid" ones, such as facts about surface form or named entities. It then tests whether the source entails each remaining fact. If no valid facts remain, a fallback step decides. The screening prompt is a 2-shot prompt refined from an existing G-eval prompt using feedback from misclassified examples.

On HalOmi's high-resource natural hallucination subtask, the agent outperforms the strongest baseline on average. This gives a model-agnostic, fact-based route to hallucination detection.

Key points

• Five chains: hallucination screening, atomic fact extraction, invalid fact removal, fact entailment reasoning, and supplementary judgment.
• The screening chain runs four independent LLM judgments at a high temperature to produce diverse reasoning. Disagreement among them triggers fact-level verification.
• There are four invalid-fact rules: facts about the sentence's language, facts about word counts or frequencies, facts with no real content, and facts naming specific entities. Entity names are excluded because transliteration breaks matching across languages.
• The fallback depends on direction. For English→HRLs, a fifth screening vote is cast and the majority decides. For HRLs→English, the SONAR cosine distance between source and translation is compared with a tuned threshold.
• On HalOmi, the agent beats the best compared method in average performance over 8 language pairs. It also exceeds the best Matthews correlation coefficient on 5 of them. The authors also estimate API cost, using ChatGPT-3.5-turbo-instruct pricing as the example.