Detecting and Mitigating Hallucinations in Machine Translation: Model Internal Workings Alone Do Well, Sentence Similarity Even Better
Paper · 2023 · David Dale, Elena Voita, Loic Barrault et al.
Summary
Neural MT systems sometimes produce output that has little or nothing to do with the input. These hallucinations are rare but badly damage user trust, and earlier detectors performed poorly on naturally occurring cases. The authors use the testbed of Guerreiro et al. (2022): a fairseq Transformer base model for German→English and its manually annotated translations. They ask how much the translation model alone can reveal. Their internal signal is ALTI+, which measures what share of each generated token is attributable to the source. Translations that draw weakly on the source are flagged as hallucinations.
ALTI+ doubles detection precision for fully detached hallucinations compared with Seq-Logprob. When used to rerank Monte Carlo dropout hypotheses, it reduces hallucinations as well as the external COMET-QE does. Among external tools, cross-lingual sentence similarity (LaBSE, LASER) and an XNLI model detect hallucinations better than quality-estimation models. LaBSE performs best overall for both detection and mitigation. The main practical value is that mitigation can work even for language pairs that lack a QE model.
Key points
- Data and model: The model is a Transformer base trained on WMT'18 De–En data without Paracrawl (5.8M pairs). The annotated set has 3415 translations: 323 hallucinations, 1044 less severe errors, and the rest correct.
- Detection AUC (all hallucinations): LaBSE scores 91.7 and XNLI 90.9. ALTI scores 84.9, Seq-Logprob 83.0 and COMET-QE 70.2. Similarity-based models reach about 80% higher precision than earlier methods.
- Human evaluation: 200 source sentences produced 602 deduplicated translation pairs, each labeled by 3 annotators. All rerankers cut the hallucination rate by roughly 2.5–3×. ALTI and COMET-QE did not differ significantly, while LaBSE was significantly better at the 5% level.
- Limitation: ALTI needs access to a Transformer model's internals, so it does not apply to statistical MT.
- Code: The code is released in facebookresearch/stopes (demo/alti/detecting_hallucinations).
📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works
🧭 Topics 1 Machine translation · 958 works
Cite
Cite as David Dale, Elena Voita, Loic Barrault and Marta R. Costa-jussà. 2023. Detecting and Mitigating Hallucinations in Machine Translation: Model Internal Workings Alone Do Well, Sentence Similarity Even Better. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
@inproceedings{dale2023detecting,
title = {{Detecting and Mitigating Hallucinations in Machine Translation: Model Internal Workings Alone Do Well, Sentence Similarity Even Better}},
author = {David Dale, Elena Voita, Loic Barrault and Marta R. Costa-jussà},
booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
year = {2023},
doi = {10.18653/v1/2023.acl-long.3},
url = {https://doi.org/10.18653/v1/2023.acl-long.3}
}References
- Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597–610. \#\# Limitations our card
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/dale-2023-detecting-mitigating-hallucinations-machine-translation/