Elamur Research⌕
☰
home / papers / 2026 / Probabilistic Bilingual Subword Segmentation with Latent Subword Alignment

Probabilistic Bilingual Subword Segmentation with Latent Subword Alignment

Paper · 2026 · Shoto Nishida, Daiki Matsui, Takashi Ninomiya et al.

Summary

Standard subword tokenizers for neural machine translation, such as BPE or SentencePiece unigram models, are trained on each language separately. They ignore the translation relationship, so a word and its translation can be split into pieces that do not correspond, which may confuse the NMT model. Nishida et al. propose a bilingual segmentation method. A SentencePiece unigram model first produces the 10 most probable segmentations for each source and target sentence. The EM algorithm then estimates latent alignment probabilities between source and target subwords. For training data, the chosen segmentation pair maximizes unigram likelihood multiplied by alignment scores. At test time, no target sentence exists, so the alignment probabilities are marginalized over target subwords to score the source candidates. The method was evaluated with Transformer base models on six language pairs and 16 translation directions. It improved BLEU in 13 of 16 tasks, with the largest gains when translating into languages written without spaces. COMET gains were less consistent. The approach adds cross-lingual consistency without requiring an NMT model to be trained inside the segmentation step.

Key points

  • Data covers six language pairs: en-ja, ja-zh, en-de, en-hi, en-id and en-th. The corpora are ASPEC, KFTT, Wikimatrix, WMT18 News Commentary v13 and scb-mt-en-th. The flores200 dev and devtest sets served for validation and testing on Wikimatrix and scb-mt-en-th.
  • Setup: SentencePiece unigram models with 16k vocabulary per language and K=L=10 candidates. Translation used Fairseq Transformer base, trained for 30 epochs and averaged over 3 seeds.
  • BLEU (SacreBLEU) improved in 13 of 16 directions, for example ASPEC en-ja 27.2→27.6 and ja-en 27.0→27.5. COMET improved in 10 of 16 directions. The authors read this as better lexical choice rather than better overall meaning.
  • Source-side segmentations stay largely the same as the baseline, while target-side segmentations change much more. The authors attribute this to an asymmetry in the alignment term of the model.
  • Limitations: the method needs sizeable parallel data and models alignments as context-independent one-to-one links. It only considers top-K candidates and raises training cost compared with conventional segmentation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Machine translation · 1,087 worksTokenization & segmentation · 119 works

Cite

Cite as Shoto Nishida, Daiki Matsui, Takashi Ninomiya, Isao Goto and Akihiro Tamura. 2026. Probabilistic Bilingual Subword Segmentation with Latent Subword Alignment. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-srw.40

@inproceedings{nishida2026probabilistic,
  title = {{Probabilistic Bilingual Subword Segmentation with Latent Subword Alignment}},
  author = {Shoto Nishida, Daiki Matsui, Takashi Ninomiya, Isao Goto and Akihiro Tamura},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-srw.40},
  url = {https://doi.org/10.18653/v1/2026.eacl-srw.40}
}

References

17 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hiroyuki Deguchi, Masao Utiyama, Akihiro Tamura, Takashi Ninomiya, and Eiichiro Sumita. 2020. Bilingual Subword Segmentation for Neural Machine Translation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4287–4297. find
  2. Tatsuya Hiraoka, Sho Takase, Kei Uchiumi, Atsushi Keyaki, and Naoaki Okazaki. 2021. Joint Optimization of Tokenization and Downstream Model. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 244–255. find
  3. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR 2015).
  4. Taku Kudo. 2018. Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75.
  5. Taku Kudo and John Richardson. 2018. Sentence-Piece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71. find
  6. Taku Kudo, Kaoru Yamamoto, and Yuji Matsumoto. 2004. Applying Conditional Random Fields to Japanese Morphological Analysis. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 230–237.
  7. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual Denoising Pre-training for Neural Machine Translation. Transactions of the Association for Computational Linguistics, 8:726–742. find
  8. Lalita Lowphansirikul, Charin Polpanumas, Attapol T Rutherford, and Sarana Nutanong. 2022. A Large English–Thai Parallel Corpus from the Web and Machine-Generated Text. Language Resources and Evaluation, 56(2):477–499. find
  9. Toshiaki Nakazawa, Manabu Yaguchi, Kiyotaka Uchimoto, Masao Utiyama, Eiichiro Sumita, Sadao Kurohashi, and Hitoshi Isahara. 2016. ASPEC: Asian Scientific Paper Excerpt Corpus. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 2204–2208.
  10. Graham Neubig. 2011. The Kyoto Free Translation Task. http://www.phontron.com/kftt. ↗
  11. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A Fast, Extensible Toolkit for Sequence Modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstr…
  12. Matt Post. 2018. A Call for Clarity in Reporting BLEU Scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191.
  13. Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022. COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages…
  14. Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2021. Wiki-Matrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages… find
  15. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725.
  16. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, and 20 o… find
  17. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nishida-2026-probabilistic-bilingual-subword-segmentation-latent/