Elamur Research⌕
☰
home / papers / 2026 / Beyond BLEU: Linguistic Invisibility and Interactional Repair Sequence in End-to-End Sign Language Translation

Beyond BLEU: Linguistic Invisibility and Interactional Repair Sequence in End-to-End Sign Language Translation

Summary

The authors contend that end-to-end sign language translation (SLT) systems can post strong BLEU/WER results while losing non-manual signals (NMS), such as mouthing, that carry meaning in real conversation. They call this effect "linguistic invisibility." To investigate it, they pair a critical survey of recent SLT pipelines with a detailed analysis of a single repair episode taken from a corpus of conversational Japanese Sign Language (JSL). The survey examines feature fusion, CTC-style alignment and loss design, judging each by whether separate articulatory channels stay detectable. In the sequence, a signer tries seven times to convey "anime." The trouble is finally resolved by changing the mouthing, while the manual sign stays almost unchanged. Kinematic measures show a clear decoupling between mouth and hand activity. Training objectives that favour overall motion similarity would likely erase this contrast. The authors therefore recommend that SLT benchmarks add multi-channel diagnostic test sets and metrics sensitive to cross-channel decoupling, so that progress reflects linguistic structure and not only sequence-level scores.

Key points

  • The case study uses a JSL conversational corpus of Deaf adult pairs from Gunma (Bono, 2017). Across seven repair trials, the signers switch between regional signs, similar signs and fingerspelling.
  • The manual form repeatedly returns to the regional sign COMIC1, while the mouthing changes from ma-n-ga to a-ni-me. This mouthing change is what finally settles the meaning.
  • The kinematic analysis compares Mouth Opening Magnitude (MAR) with Right-Hand Movement Energy. From 0 s to 7.11 s, the hand oscillates steadily while the mouth shows strong, pulsing activity.
  • The survey centres on state-of-the-art gloss-free and end-to-end SLT benchmarked on PHOENIX-2014T and How2Sign. It also traces the origins of BLEU and WER as evaluation standards.
  • The authors acknowledge that the evidence comes from a single sequence and is not statistically generalizable. They also note that MAR and hand energy are only proxies that leave out eye gaze, facial expression and head movement.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3drmbqqsx7a8

@inproceedings{wang2026beyond,
  title = {{Beyond BLEU: Linguistic Invisibility and Interactional Repair Sequence in End-to-End Sign Language Translation}},
  author = {Wang, Zirui and Bono, Mayumi},
  booktitle = {Proceedings of the LREC 2026 12th Workshop on the Representation and Processing of Sign Languages: Language in Motion},
  year = {2026},
  doi = {10.63317/3drmbqqsx7a8},
  url = {https://doi.org/10.63317/3drmbqqsx7a8}
}

References

18 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bono, M. 2017. Improvisational signing in sign language interaction through a lens of repair sequence. The Japanese Journal of Language in Society, 19(2), 20–31. DOI: 10.19024/jajls.19.2_59 ↗
  2. Bono, M. 2026. Multimodal corpora. In H. Nesi and P. Milin (eds.), International Encyclopedia of Language and Linguistics, 3rd ed. Elsevier. find
  3. Bono, M., Okada, T., Kikuchi, K., Sakaida, R., Skobov, V., Miyao, Y., and Osugi, Y. 2023. Utterance unit annotation for the Japanese Sign Language Dialogue Corpus: Towards a method for detecting interactional boundaries in spontaneous sign language dialogue. In Ella Wehrmeyer (ed.), Advances in Sign Language Corpus Linguistics, SCL 108, pp. 353–382. John Benjamins. URL: https://cir.nii.ac.jp/crid/13605802306328592 64 Bragg, D., Koller, O., Bellard, M., Berke, L., Boudreault, P., Braffort, A., Caselli, N., Huenerfauth, M., Kacorri, H., Verhoef, T., Vogler, C., and Morris, M. R. 2019. Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective. In Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’19), pages 16–31, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3308561.3353774 ↗
  4. Camgoz, N. C., Koller, O., Hadfield, S., and Bowden, R. 2018. Neural Sign Language Translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). find
  5. De Meulder, M. 2021. Is “Good Enough” Good Enough? Ethical and Responsible Development of Sign Language Technologies. In Proceedings of Machine Translation Summit XVIII: Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL). URL: https://aclanthology.org/2021.mtsummit-at4ssl.2/ ↗
  6. Duarte, A., et al. 2021. How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. DOI: 10.1109/CVPR46437.2021.00276 ↗
  7. Forster, J., Schmidt, C., Hoyoux, T., Koller, O., Zelle, U., Piater, J., and Ney, H. 2012. RWTH-PHOENIX-Weather: A Large Vocabulary Sign Language Recognition and Translation Corpus. In Proceedings of LREC 2012, Istanbul, Turkey. URL: https://aclanthology.org/L12-1503/ ↗
  8. Graves, A., Fernández, S., Gomez, F., and Schmidhuber, J. 2006. Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks. In Proceedings of ICML 2006. find
  9. Hwang, E. J., Cho, S., Lee, J., and Park, J. C. 2025. An Efficient Gloss-Free Sign Language Translation Using Spatial Configurations and Motion Dynamics with LLMs. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3901–3920, Albuquerque, New Mexico, USA. Association for Computational Linguistics. find
  10. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of ACL 2002. find our card
  11. Popović, M. 2015. chrF: character n-gram Fscore for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation. find
  12. Regmi, B. S. 2025. Mouth Movement Classification in Japanese Sign Language. Internship Report, National Institute of Informatics, Tokyo, Japan. find
  13. Sandermann, T. H. 2025. Temporal Mouth Movement Segmentation in Japanese Sign Language. Internship Report, National Institute of Informatics, Tokyo, Japan. find
  14. Skobov, V. and Bono, M. 2023. Making Body Movement in Sign Language Corpus Accessible for Linguists and Machines with Three-Dimensional Normalization of MediaPipe. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1913–1926, Singapore. Association for Computational Linguistics. ↗
  15. Snover, M., Dorr, B., Schwartz, R., Micciulla, L., and Makhoul, J. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of AMTA. find our card
  16. Wong, R., Camgoz, N. C., and Bowden, R. 2024. Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation. In Proceedings of the International Conference on Learning Representations. ↗
  17. Yin, K., Kordjamshidi, P., and Roth, D. 2021. Including Signed Languages in Natural Language Processing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, pages 5774–5787. ↗
  18. Zhou, B., Zhang, S., Chen, X., Liu, M., and Wu, Y. 2023. Gloss-Free Sign Language Translation: Improving from Visual-Language Pretraining. In Proceedings of the IEEE/CVF International Conference on Computer Vision. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-beyond-bleu-linguistic-invisibility-interactional/