Elamur Research⌕
☰
home / papers / 2009 / Joshua: An Open Source Toolkit for Parsing-Based Machine Translation

Joshua: An Open Source Toolkit for Parsing-Based Machine Translation

Summary

Parsing-based statistical MT (Hiero-style hierarchical models and other syntax-driven approaches) had advanced rapidly. However, the strongest systems ran on private, purpose-built code, so other researchers struggled to reproduce or compare results. The authors release Joshua, an open source Java toolkit meant to do for SCFG-based translation what Moses did for phrase-based MT. It implements the core machinery from Chiang (2007): CKY chart parsing over synchronous context-free grammars, n-gram language model integration, beam and cube pruning, and k-best extraction from hypergraphs. Supporting components include test-set-driven training data sub-sampling, suffix-array grammar extraction, several language model back-ends, parallel and distributed decoding, and a MERT module. The design emphasises three goals: extensibility through Java interfaces, a coherent end-to-end pipeline, and scalability. For the WMT09 French-English task, the authors also built a new web-crawled parallel corpus of 21.9M sentence pairs. The baseline scored 25.92 uncased BLEU, and adding deterministic annealing with variational decoding raised this to 26.52. The release lowers the entry cost for syntax-based MT research.

Key points

  • Sub-sampling. The method collects every n-gram up to length 10 from the dev and test sets. It then keeps a training pair if the pair's length ratio is typical and its source side contains an n-gram still seen fewer than k times. With k = 20, this selected 1.5M of 23M sentence pairs, about 30M words per language.
  • Training corpus. The French-English crawl covered Canadian government, EU and international organisation sites, roughly 40M files and over 1TB. URL-rewriting heuristics yielded 2M paired documents, which were sentence-aligned with Moore's (2002) aligner. Pairs longer than 100 words (630K) were removed, leaving 21.9M pairs. This corpus was shared with WMT09 participants and used alongside Europarl v4 (about 1.4M pairs).
  • Language models. Joshua offers three back-ends:
    • a pure-Java reader for ARPA backoff models, so SRILM is not required;
    • a native bridge to SRILM, which scales better;
    • a Bloom filter LM (Talbot and Osborne, 2007).
  • WMT09 setup. The submission used a 5-gram SRILM model trained on 21.2M English news and blog sentences with a 500K-word vocabulary. Tuning used the WMT08 News Test Set (2,051 sentences), and the test set had 2,525 segments.
  • Results (uncased BLEU-4).
    • Baseline: 25.92
    • MBR rescoring of the top 300 outputs: 26.16
    • Deterministic annealing: 25.98
    • Deterministic annealing plus variational decoding: 26.52
    • The MERT module is also released separately as Z-MERT.

Interesting ideas

  • I1. Test-set-aware data selection with a coverage cap. A sentence is kept only while some n-gram from the target test material remains under-represented. This greedy filter could shrink huge, noisy Ukrainian parallel or web corpora into a manageable, task-focused training set.
  • I2. Extracting only the rules a test set needs. A source-side suffix array lets the system pull just the grammar rules required for specific test sentences, instead of precomputing everything. This avoids full re-extraction when you change tokenization or segmentation, which is valuable when experimenting with morphologically rich languages like Ukrainian.
  • I3. Interfaces plus abstract base classes, with components that run standalone. Each extensible part is defined as an interface and ships with a basic default implementation. Researchers can then swap in one piece without touching the rest, and the same pattern suits any shared NLP pipeline in our library.
  • I4. Pure-language fallback for native dependencies. Shipping a simple Java LM next to the faster SRILM bridge lets newcomers try the tool without compiling native code. Offering an easy path alongside a scalable path is a good onboarding pattern for internal tools.
  • I5. Finding parallel documents through URL patterns. Mapping French URLs to English counterparts on bilingual sites cheaply produced millions of document pairs before sentence alignment. A similar trick could work on bilingual Ukrainian/English official or organisational websites.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

85 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 80 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{li2009joshua,
  title = {{Joshua: An Open Source Toolkit for Parsing-Based Machine Translation}},
  author = {Li, Zhifei and Callison-Burch, Chris and Dyer, Chris and Khudanpur, Sanjeev and Schwartz, Lane and Thornton, Wren and Weese, Jonathan and Zaidan, Omar},
  booktitle = {Proceedings of the Fourth Workshop on Statistical Machine Translation},
  year = {2009},
  url = {https://aclanthology.org/W09-0424/}
}

References

25 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chris Callison-Burch, Colin Bannard, and Josh Schroeder. 2005. Scaling phrase-based statistical machine translation to larger corpora and longer phrases. In Proceedings of ACL. find
  2. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2008. Further meta-evaluation of machine translation. In Proceedings of the Third Workshop on Statistical Machine Translation (WMT08). find our card
  3. Chris Callison-Burch. 2009. A 10 9 word parallel corpus. In preparation. find
  4. David Chiang. 2007. Hierarchical phrase-based translation. Computational Linguistics, 33(2):201–228. ↗ our card
  5. Jason Eisner. 2003. Learning non-isomorphic tree mappings for machine translation. In Proceedings of ACL. find our card
  6. Michel Galley, Jonathan Graehl, Kevin Knight, Daniel Marcu, Steve DeNeefe, Wei Wang, and Ignacio Thayer. 2006. Scalable inference and training of context-rich syntactic translation models. In Proceedings of the ACL/Coling. find our card
  7. Liang Huang and David Chiang. 2005. Better k-best parsing. In Proceedings of the International Workshop on Parsing Technologies. find
  8. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of HLT/NAACL. ↗
  9. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, , and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the ACL-2007 Demo and Poster Sessions. find
  10. Philipp Koehn. 2005. A parallel corpus for statistical machine translation. In Proceedings of MT-Summit, Phuket, Thailand. find
  11. Shankar Kumar and William Byrne. 2004. Minimum bayes-risk decoding for statistical machine translation. In Proceedings of HLT/NAACL. find our card
  12. Zhifei Li and Sanjeev Khudanpur. 2008a. Large-scale discriminative n-gram language models for statistical machine translation. In Proceedings of AMTA. find
  13. Zhifei Li and Sanjeev Khudanpur. 2008b. A scalable decoder for parsing-based machine translation with equivalent language model state maintenance. In In Proceedings Workshop on Syntax and Structure in Statistical Translation. find
  14. Zhifei Li, Chris Callison-Burch, Sanjeev Khudanpur, and Wren Thornton. 2009a. Decoding in joshua: Open source, parsing-based machine translation. The Prague Bulletin of Mathematical Linguistics, 91:47–56. ↗
  15. Zhifei Li, Jason Eisner, and Sanjeev Khudanpur. 2009b. Variational decoding for statistical machine translation. In preparation. find
  16. Yang Liu, Qun Liu, and Shouxun Lin. 2006. Tree-to-string alignment templates for statistical machine translation. In Proceedings of the ACL/Coling. find
  17. Adam Lopez. 2007. Hierarchical phrase-based translation with suffix arrays. In Proceedings of EMNLP-CoLing. find our card
  18. Robert C. Moore. 2002. Fast and accurate sentence alignment of bilingual corpora. In Proceedings of AMTA. find
  19. Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51. ↗
  20. Franz Josef Och. 2003. Minimum error rate training for statistical machine translation. In Proceedings of ACL. find
  21. Chris Quirk, Arul Menezes, and Colin Cherry. 2005. Dependency treelet translation: Syntactically informed phrasal smt. In Proceedings of ACL. find our card
  22. David A. Smith and Jason Eisner. 2006. Minimum risk annealing for training log-linear models. In Proceedings of the ACL/Coling. find
  23. Andreas Stolcke. 2002. SRILM - an extensible language modeling toolkit. In Proceedings of the International Conference on Spoken Language Processing, Denver, Colorado, September. find
  24. David Talbot and Miles Osborne. 2007. Randomised language modelling for statistical machine translation. In Proceedings of ACL. find
  25. Omar F. Zaidan. 2009. Z-MERT: A fully configurable open source tool for minimum error rate training of machine translation systems. The Prague Bulletin of Mathematical Linguistics, 91:79–88. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/li-2009-joshua-open-source-toolkit-parsing/