Elamur Research⌕
☰
home / papers / 2024 / Improving Language Models Trained on Translated Data with Continual Pre-Training and Dictionary Learning Analysis

Improving Language Models Trained on Translated Data with Continual Pre-Training and Dictionary Learning Analysis

Paper · 2024 · Sabri Boughorbel, Md Rizwan Parvez and Majd Hawasly

Summary

Arabic and other lower-resource languages often build training corpora by machine-translating English text. That approach is expensive at scale, imports cultural assumptions from the source, and degrades data when translation quality is mediocre. The authors study this using TinyStories, a synthetic English corpus of simple children's stories. They translated it into Arabic with the open NLLB-3B model, chosen deliberately to represent medium-quality translation. On this data they trained a family of small Llama-2-style story generators of roughly 1M to 33M parameters. The resulting models showed weak spots: English-centric character names and malformed dialogue attribution. To repair them, the authors ran a short continual pre-training phase on 20K Arabic stories written directly by Command R+, about 1% of the original data volume. They measured the effect in two ways: GPT-4 scoring of story completions, and sparse auto-encoders trained on model activations. The interpretability analysis showed that internal features shifted toward correct dialogue tagging and toward Arabic names. The broader lesson is that a small, clean, native-language top-up can partly undo damage from cheap translated data.

Key points

  • Data. About 2.2M TinyStories were translated with NLLB-3B, which runs on a single 16GB GPU. Another 20K native Arabic stories were synthesized with Command R+, and both sets are planned for open release.
  • Evaluation. The 44 English story-completion prompts were translated with GPT-4, giving a higher-quality test set that is out of distribution relative to the NLLB training text. GPT-4 then graded grammar, creativity and consistency on a 0–10 scale, averaged over two samples per prompt at temperature 1.
  • Model setup.
    • Architecture follows the original TinyStories configurations but uses Llama-2 blocks instead of GPT-NeoX.
    • A dedicated Arabic BPE tokenizer was trained, and vocabularies of 8k, 16k, 32k and 85k were compared on a 33M reference model; 32k scored best.
    • Training used AdamW with a constant learning rate of 5e-5.
  • Translation noise. Models trained on the translated Arabic data converged to a higher loss than English-trained ones, which the authors attribute partly to noise introduced by translation.
  • Cultural bias. Sparse auto-encoders were trained on the model 2L-33M-ar, and token-set enrichment (TSEA) compared two sets of 600 English and 600 Arabic first names.
    • Before continual pre-training, many features showed a gap of at least 2 points favoring English names.
    • After continual pre-training (2L-33M-ar-CP), only one feature kept such a gap.

Interesting ideas

  • I1. Instead of re-curating a huge noisy corpus, the authors train on it first and then fix specific defects with a tiny, carefully produced native dataset; this is cheap and targets known failure modes directly.
  • I2. The test prompts were translated by a stronger system than the training data, so evaluation is not contaminated by the same translation artifacts; this kind of distribution separation is useful whenever training data is synthetic.
  • I3. Enrichment over hand-defined token sets (for example, two lists of names) gives a targeted, quantitative probe of a learned bias, rather than relying on anecdotes from samples.
  • I4. The authors picked one concrete construction damaged by translation, a person's name closing a sentence in dialogue attribution, and tracked it before and after repair.
  • I5. Vocabulary size was tuned by training the same small reference model with several tokenizers and comparing downstream judge scores, not intrinsic statistics.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): After training gparse or the perceptron parser on large silver data from witness parsers, finish with a short pass on a small hand-verified or teacher-checked set focused on the constructions the silver data gets wrong.
  • I2 (learn): Build evaluation sets with a different, stronger pipeline than the one producing silver training trees, so gains are not just agreement with the same systematic errors.
  • I3 (tokens): Define token sets (foreign versus English names, rare versus common lemmas) and check whether tagger or morphology feature weights fire unevenly across them, as a negative-control style bias audit.
  • I4 (frames): Speaker attribution in quoted speech is exactly where pragmatics assigns speech acts, so a dedicated probe set of dialogue tags could check that prag attaches the right speaker after any data or parser change.
  • I5 (tokens): Choose tokenizer or lemma-inventory settings in crate en by downstream effect on parsing and frame accuracy with a fixed small model, rather than by coverage numbers alone.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hamad bin Khalifa UniversityDoha · Qatar141 works · 14,175 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 works

Cite

Cite as Sabri Boughorbel, Md Rizwan Parvez and Majd Hawasly. 2024. Improving Language Models Trained on Translated Data with Continual Pre-Training and Dictionary Learning Analysis. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.7

@inproceedings{boughorbel2024improving,
  title = {{Improving Language Models Trained on Translated Data with Continual Pre-Training and Dictionary Learning Analysis}},
  author = {Sabri Boughorbel, Md Rizwan Parvez and Majd Hawasly},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.7},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.7}
}

References

46 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219. ↗
  2. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245. ↗
  3. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. AraGPT2: Pre-trained transformer for Arabic language generation. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 196–207, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. ↗
  4. Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. 2022. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint arXiv:2204.06745. ↗
  5. Joseph Bloom. 2024. Open source sparse autoencoders for all residual stream layers of gpt2 small. https://www.alignmentforum.org/posts/f9E gfLSurAiqRJySD/open-source-sparse-autoe ncoders-for-all-residual-stream. ↗
  6. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al. 2023. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, page 2. find
  7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  8. Ernie Chang, Hui-Syuan Yeh, and Vera Demberg. 2021. Does the order of training samples matter? improving neural data-to-text generation with curriculum learning. arXiv preprint arXiv:2102.03554. ↗
  9. Together Computer. 2023. Redpajama: an open dataset for training large language models.
  10. Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. ↗
  11. Ronen Eldan and Yuanzhi Li. 2023. Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.07759. ↗
  12. Ali Elfilali, Hamza Alobeidli, Clémentine Fourrier, Basma El Amel Boussaha, Ruxandra Cojocaru, Nathan Habib, and Hakim Hacid. 2024. Open arabic llm leaderboard. https://huggingface.co/s paces/OALL/Open-Arabic-LLM-Leaderboard. ↗
  13. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. 2022a. Toy models of superposition. Transformer Circuits Thread. Https://transformer-circuits.pub/2022/toy_model/index.html. find
  14. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. 2022b. Toy models of superposition. arXiv preprint arXiv:2209.10652. ↗
  15. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1:1. find
  16. AbdelRahim Elmadany, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. Octopus: A multitask model and toolkit for Arabic natural language generation. In Proceedings of ArabicNLP 2023, pages 232–243, Singapore (Hybrid). Association for Computational Linguistics.
  17. Kshitij Gupta, Benjamin Thérien, Adam Ibrahim, Mats L Richter, Quentin Anthony, Eugene Belilovsky, Irina Rish, and Timothée Lesort. 2023. Continual pre-training of large language models: How to (re) warm your model? arXiv preprint arXiv:2308.04014. ↗
  18. Oskar Holmström, Jenny Kunz, and Marco Kuhlmann. 2023. Bridging the resource gap: Exploring the efficacy of English and multilingual LLMs for Swedish. In Proceedings of the Second Workshop on Resources and Representations for Under-Resourced Languages and Domains (RESOURCEFUL-2023), pages 92–110, Tórshavn, the Faroe Islands. Association for Computational Linguistics.
  19. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Ziche Liu, et al. 2023. Acegpt, localizing large language models in arabic. arXiv preprint arXiv:2309.12053. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825. ↗
  20. Joseph Bloom Johnny Lin. 2024. Announcing neuronpedia: Platform for accelerating research into sparse autoencoders. https://www.lesswrong.com/posts/BaEQoxHhWPrkinmxd/announcing -neuronpedia-as-a-platform-to-accelerat e-research. ↗
  21. Johnny Lin Joseph Bloom. 2024. Understanding sae features with the logit lens. https://www.lesswr ong.com/posts/qykrYY6rXXM7EEs8Q/understa nding-sae-features-with-the-logit-lens. ↗
  22. Moussa Kamal Eddine, Nadi Tomeh, Nizar Habash, Joseph Le Roux, and Michalis Vazirgiannis. 2022. AraBART: a pretrained Arabic sequence-to-sequence model for abstractive summarization. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 31– 42, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  23. Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. 2023. Continual pre-training of language models. arXiv preprint arXiv:2302.03241. ↗
  24. Anis Koubaa, Adel Ammar, Lahouari Ghouti, Omar Najar, and Serry Sibaee. 2024. Arabiangpt: Native arabic gpt-based large language model. Preprint, arXiv:2402.15313. ↗
  25. Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat. 2024. Madlad-400: A multilingual and document-level large audited dataset. Advances in Neural Information Processing Systems, 36.
  26. Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463. ↗
  27. Hanxiao Liu, Zihang Dai, David So, and Quoc V Le. 2021. Pay attention to mlps. Advances in neural information processing systems, 34:9204–9215. ↗
  28. Aleksandar Makelov, George Lange, and Neel Nanda. 2024. Towards principled evaluations of sparse autoencoders for interpretability and control. arXiv preprint arXiv:2405.08366. ↗
  29. Sachin Mehta, Mohammad Sekhavat, Qingqing Cao, Max Horton, Yanzi Jin, Frank Sun, Iman Mirzadeh, Mahyar Najibikohnehshahri, Dmitry Belenko, Peter Zatloukal, and Mohammad Rastegari. 2024. Openelm: An efficient language model family with open training and inference framework. ↗
  30. Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, et al. 2023. Seallms–large language models for southeast asia. arXiv preprint arXiv:2312.00738. ↗
  31. Guilherme Penedo, Hynek Kydlíček, Leandro von Werra, and Thomas Wolf. 2024. Fineweb. find
  32. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  33. Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, Janos Kramar, Rohin Shah, and Neel Nanda. 2024. Improving dictionary learning with gated sparse autoencoders. arXiv preprint arXiv:2404.16014. ↗
  34. Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. 2024. Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems, 36.
  35. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong Han, Sondos Mahmoud Bsharat, Alham Fikri Aji, Zhiqiang Shen, Zhengzhong Liu, Natalia Vassilieva, Joel Hestness, Andy Hock, Andrew Feldman, Jonathan Lee, Andrew Jackson, Hector Xuguang Ren, Preslav Nakov, Timothy Baldwin, and Eric Xing. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. Preprint, arXiv:2308.16149. ↗
  36. Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. 2024. Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. arXiv preprint. ↗
  37. Aravind Subramanian, Pablo Tamayo, Vamsi K Mootha, Sayan Mukherjee, Benjamin L Ebert, Michael A Gillette, Amanda Paulovich, Scott L Pomeroy, Todd R Golub, Eric S Lander, et al. 2005. Gene set enrichment analysis: a knowledge-based approach for interpreting genome-wide expression profiles. Proceedings of the National Academy of Sciences, 102(43):15545–15550. ↗
  38. Brian Thompson, Mehak Preet Dhaliwal, Peter Frisch, Tobias Domhan, and Marcello Federico. 2024. A shocking amount of the web is machine translated: Insights from multi-way parallelism. Preprint, arXiv:2401.05749. ↗
  39. Jörg Tiedemann, Mikko Aulamo, Daria Bakshandaeva, Michele Boggia, Stig-Arne Grönroos, Tommi Nieminen, Alessandro Raganato, Yves Scherrer, Raul Vazquez, and Sami Virpioja. 2023. Democratizing neural machine translation with opus-mt. Language Resources and Evaluation, pages 1–43. ↗
  40. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, ..., and Thomas Wolf. 2023. Bloom: A 176b-parameter open-access multilingual language model. Preprint, arXiv:2211.05100. ↗
  41. Yong Xie, Karan Aggarwal, and Aitzaz Ahmad. 2023. Efficient continual pre-training for building domain specific large language models. arXiv preprint arXiv:2311.08545. ↗
  42. Biao Zhang and Rico Sennrich. 2019. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32. ↗
  43. Chen Zhang, Dawei Song, Zheyu Ye, and Yan Gao. 2023. Towards the law of capacity gap in distilling language models. arXiv preprint arXiv:2311.07052. ↗
  44. Mike Zhang and Antonio Toral. 2019. The effect of translationese in machine translation test sets. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), pages 73– 81, Florence, Italy. Association for Computational Linguistics.
  45. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36.
  46. Tianyi Zhou, Shengjie Wang, and Jeffrey Bilmes. 2020. Curriculum learning by dynamic instance hardness. Advances in Neural Information Processing Systems, 33:8602–8613. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/boughorbel-2024-improving-language-models-trained-translated/