Elamur Research⌕
☰
home / papers / 2023 / Lost in Translation: Large Language Models in Non-English Content Analysis

Lost in Translation: Large Language Models in Non-English Content Analysis

Paper · 2023 · Gabriel Nicholas and Aliya Bhatia

Summary

Automated tools that moderate, search, and generate online text work much better in English than in the roughly 7,000 other languages people use. This CDT policy report by Nicholas and Bhatia asks whether multilingual language models close that divide when used for content analysis. These models are trained on many languages together, so that patterns learned from data-rich languages can carry over to data-poor ones. The authors run no new experiments and release no dataset. Instead, they synthesize technical research and industry practice. They explain how large language models are built, define the "resourcedness gap" in training data, and list the weaknesses of multilingual models. Their main finding is that these models still lean heavily on English and perform worst for low-resource languages. They may also carry English-centric meanings into other cultures and fail in ways that are hard to trace. Filling gaps with machine-translated text can add further errors. The report ends with recommendations for companies, researchers and funders, and governments. This matters because platforms already use such models to moderate speech, which affects non-English speakers' expression and access to services.

Key points

  • Multilingual models cited include Meta's XLM-RoBERTa (XLM-R), trained on 100 languages, and Google's mBERT, trained on 104. English-focused counterparts include GPT-4, LLaMa, and PaLM.
  • Real deployments include Meta's harmful-content detection across more than 100 languages, Alphabet's Perspective API for toxicity in 18 languages, and Bumble's filtering of unwanted sexual messages.
  • Languages are grouped by resourcedness:
    • High: English (far ahead), Spanish, Chinese, German.
    • Medium: Russian, Hebrew, Vietnamese.
    • Low: Amharic, Cherokee, Haitian Creole. Their data is often noisy or limited to narrow sources such as religious texts and Wikipedia.
  • The report illustrates cross-lingual value transfer with the word "dove." In English it suggests peace, but the Basque equivalent, "uso," can be used as an insult.
  • The recommendations are:
    • Companies should disclose where and in which languages they use these models, keep tasks narrow, and provide human review.
    • Funders should support local-language research communities.
    • Governments should not use these models for unsupervised high-stakes decisions or legally mandate them for moderation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 works

🧭 Topics 1Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 works

Cite

Cite as Gabriel Nicholas and Aliya Bhatia. 2023. Lost in Translation: Large Language Models in Non-English Content Analysis. arXiv:2306.07377.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.07377 · arXiv 2306.07377

@misc{nicholas2023lost,
  title = {{Lost in Translation: Large Language Models in Non-English Content Analysis}},
  author = {Gabriel Nicholas and Aliya Bhatia},
  year = {2023},
  doi = {10.48550/arxiv.2306.07377},
  eprint = {2306.07377},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.07377}
}

References

146 references, 42 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abid, A., Farooqi, M., & Zou, J. (2021). Large language models associate Muslims with violence. Nature Machine Intelligence, 3(6), Article 6. [perma.cc/HK4B-3AAQ] find
  2. ACL 2022 Theme Track: “Language Diversity: from Low-Resource to Endangered Languages.” ACL. [perma.cc/F2YW-QZBP] find
  3. ACL Rolling Review Dashboard. (2022). Papers Mentioning \>0 Languages. [perma.cc/EQU9-5CWQ] find
  4. Agerri, R., Vicente, I. S., Campos, J. A., Barrena, A., Saralegi, X., Soroa, A., & Agirre, E. (2020). Give your Text Representation Models some Love: The Case for Basque. Proceedings of the 12th Conference on Language Resources and Evaluation, 4781–4788. [perma.cc/R2DA-GGQZ] find
  5. Alyafeai, Z., & Al-Shaibani, M. (2020). ARBML: Democratizing Arabic Natural Language Processing Tools. Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS), 8–13. [perma.cc/4TFY-E9EJ]
  6. Amer, M. (2022, July 13). Large Language Models and Where to Use Them: Part 2. Cohere. [perma.cc/CRT5-HDX8] find
  7. AmericasNLP. (2022, December 7). Second Workshop on NLP for Indigenous Languages of the Americas (AmericasNLP). [perma.cc/SC88-9WGF] find
  8. Amrute, S., Singh, R., & Guzmán, R. L. (2022). A Primer on AI in/from the Majority World. Data & Society. [perma.cc/SR8B-J2L9] find
  9. Antoun, W., Baly, F., & Hajj, H. (2020). AraBERT: Transformer-based Model for Arabic Language Understanding. Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, 9–15. [perma.cc/X5VJ-JKXQ]
  10. Artetxe, M., Labaka, G., & Agirre, E. (2020). Translation Artifacts in Cross-lingual Transfer Learning. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 7674–7684. [perma.cc/MZY5-DL83]
  11. Artetxe, M., Ruder, S., & Yogatama, D. (2020). On the Cross-lingual Transferability of Monolingual Representations. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 4623–4637. [perma.cc/7WMN-5QPR]
  12. Artetxe, M., & Schwenk, H. (2019). Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond. Transactions of the Association for Computational Linguistics, 7, 597–610. [perma.cc/ LB6R-GH9K]
  13. Article 19. (2022). Bridging the Gap: Local voices in content moderation. Bosnia and Herzegovina. [perma.cc/ASU5-ST4N] find
  14. Avaaz. (2019). Megaphone for Hate: Disinformation and Hate Speech on Facebook During Assam’s Citizenship Count. Avaaz. [perma.cc/5MXS-7P7N] find
  15. Basile, V., Maro, M. D., Croce, D., & Passaro, L. (2020, December 17). find
  16. EVALITA 2020: Overview of the 7th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Seventh Evaluation Campaign of Natural Language Processing and Speech Tools for Italian, Online. [perma.cc/76EK-EJQ8]
  17. Belloni, M. (2021, December 8). Multilingual message content moderation at scale. Bumble Tech. [perma.cc/ RL2A-L2BD] find
  18. Bender, E. (2019, September 15). The \#BenderRule: On Naming the Languages We Study and Why It Matters. The Gradient. [perma.cc/J3ZM-A5UP] find
  19. Bender, E., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? . Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610–623. [perma.cc/3KLC-TBUY] find
  20. 🦜 Bender, E., & Koller, A. (2020). Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5185–5198. [perma.cc/TN3W-5NTC]
  21. Bergman, A., & Diab, M. (2022). Towards Responsible Natural Language Annotation for the Varieties of Arabic. Findings of the Association for Computational Linguistics: ACL 2022, 364–371. [perma.cc/Q37M-8F2Y]
  22. BigScience Workshop, Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F., Gallé, M., Tow, J., Rush, A. M., Biderman, S., Webson, A., Ammanamanchi, P. S., Wang, T., Sagot, B., Muennighoff, N., … Wolf, T. (2023). BLOOM: A 176B-Parameter Open-Acc… find
  23. Birhane, A., Kalluri, P., Card, D., Agnew, W., Dotan, R., & Bao, M. (2022). The Values Encoded in Machine Learning Research. 2022 ACM Conference on Fairness, Accountability, and Transparency, 173–184. [perma.cc/9GNB-JHQ5] find
  24. Birhane, A., & Prabhu, V. U. (2021). Large image datasets: A pyrrhic win for computer vision? 2021 IEEE Winter Conference on Applications of Computer Vision, 1536–1546. [perma.cc/Q8LP-THYK] find
  25. Bizzoni, Y., Juzek, T. S., España-Bonet, C., Dutta Chowdhury, K., van Genabith, J., & Teich, E. (2020). How Human is Machine Translationese? Comparing Human and Machine Translations of Text and Speech. Proceedings of the 17th International Conference on Spoken Language Translation, 280–290. [perma.… find
  26. Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N., Chen, A., Creel, K., Davis, J. Q., Demszky, D., … Liang, P. (2021). On the Opportunities and Risks…
  27. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., … Amodei, D. (2020). Language Models are Few-Shot Learners.…
  28. Cahyawijaya, S., Lovenia, H., Aji, A. F., Winata, G. I., Wilie, B., Mahendra, R., Wibisono, C., Romadhony, A., Vincentio, K., Koto, F., Santoso, J., Moeljadi, D., Wirawan, C., Hudi, F., Parmonangan, I. H., Alfina, I., Wicaksono, M. S., Putra, I. F., Rahmadani, S., … Purwarianti, A. (2022). NusaCrowd… find
  29. Callahan, E. S., & Herring, S. C. (2011). Cultural bias in Wikipedia content on famous persons. Journal of the American Society for Information Science and Technology, 62(10), 1899–1915. [perma.cc/2S8K-YEJK] find
  30. Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., & Zhang, C. (2023, February 1). Quantifying Memorization Across Neural Language Models. The Eleventh International Conference on Learning Representations. [perma.cc/678U-9PAQ] find
  31. Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., Oprea, A., & Raffel, C. (2021). Extracting Training Data from Large Language Models (arXiv:2012.07805). arXiv. [perma.cc/58MA-VWRZ]
  32. Caswell, I., Breiner, T., van Esch, D., & Bapna, A. (2020). Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus. Proceedings of the 28th International Conference on Computational Linguistics, 6588–6608. [perma.cc/8RFD-DTUK]
  33. Chi, E. A., Hewitt, J., & Manning, C. D. (2020). Finding Universal Grammatical Relations in Multilingual BERT. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5564–5577. [perma.cc/8LNR-VNY9] find
  34. Choi, H., Kim, J., Joe, S., Min, S., & Gwon, Y. (2021). Analyzing Zero-shot Cross-lingual Transfer in Supervised NLP Tasks (arXiv:2101.10649). arXiv. [perma.cc/NEB9-8THZ] — [arXiv:2101.10649](https://arxiv.org/abs/2101.10649) find
  35. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., … Fiedel, N. (2022). PaLM: Scaling Language Modeling with Path…
  36. Christian, J. (2018, July 20). Why Is Google Translate Spitting Out Sinister Religious Prophecies? Vice. [perma. cc/8YQU-NUFM] find
  37. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020). Unsupervised Cross-lingual Representation Learning at Scale. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 8440–8451.… find
  38. Conneau, A., & Lample, G. (2019). Cross-lingual Language Model Pretraining. Advances in Neural Information Processing Systems, 32. [perma.cc/N7EE-JM83] find
  39. Conneau, A., Wu, S., Li, H., Zettlemoyer, L., & Stoyanov, V. (2020). Emerging Cross-lingual Structure in Pretrained Language Models. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 6022–6034. [perma.cc/3NHR-G7Y4]
  40. Corradi, A. (2017, April 25). The Linguistic Colonialism of English. Brown Political Review. [perma.cc/5M3M-9EMN] find
  41. Corvey, W. (2014). Low Resource Languages for Emergent Incidents. Defense Advanced Research Projects Agency. [perma.cc/4FDR-M3YC] find
  42. Crawford, K. (2021). Atlas of AI: Power, politics, and the planetary costs of artificial intelligence. find
  43. Yale University Press. de Varda, A. G., & Marelli, M. (2023). Data-driven Cross-lingual Syntax: An Agreement Study with Massively Multilingual Models. Computational Linguistics, 1–39. [perma.cc/7LQP-EEBQ] de Vries, W., van Cranenburgh, A., Bisazza, A., Caselli, T., van Noord, G., & Nissim, M. (2019)…
  44. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long…
  45. Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., & Gardner, M. (2021). Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 1286–1305. [per… find
  46. Duarte, N., Llansó, E., & Loup, A. C. (2017). Mixed Messages? The Limits of Automated Social Media Content Analysis. Center for Democracy & Technology. [perma.cc/9BRH-5ZZN] find
  47. Dulhanty, C., Deglint, J. L., Daya, I. B., & Wong, A. (2019, November 26). Taking a Stance on Fake News: Towards Automatic Disinformation Assessment via Deep Bidirectional Transformer Language Models for Stance Detection. find
  48. NeurIPS 2019, Vancouver. [perma.cc/P5JD-5AD9] find
  49. Ebers, M., Poncibò, C., & Zou, M. (Eds.). (2022). Contracting and Contract Law in the Age of Artificial Intelligence. Hart Publishing. [perma.cc/G4XR-VYNL] find
  50. Eronen, J., Ptaszynski, M., & Masui, F. (2023). Zero-shot cross-lingual transfer language selection using linguistic similarity. Information Processing & Management, 60(3), 103250. [perma.cc/S78N-C9MR] find
  51. Ethnologue. (2023a). Assamese. Ethnologue, Languages of the World. [perma.cc/BE78-H3PN] find
  52. Ethnologue. (2023b). Statistics. Ethnologue, Languages of the World. [perma.cc/H27U-44TK] find
  53. Ganguli, D., Hernandez, D., Lovitt, L., DasSarma, N., Henighan, T., Jones, A., Joseph, N., Kernion, J., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Elhage, N., Showk, S. E., Fort, S., Hatfield-Dodds, Z., Johnston, S., … Clark, J. (2022). Predictability and Surprise in Large Gene… find
  54. Golebiewski, M., & boyd, danah. (2018). Data Voids: Where Missing Data Can Easily Be Exploited. Data & Society. [perma.cc/HE5A-7QTJ] find
  55. Góngora, S., Giossa, N., & Chiruzzo, L. (2021). Experiments on a Guarani Corpus of News and Social Media. Proceedings of the First Workshop on Natural Language Processing for Indigenous Languages of the Americas, 153–158. [perma.cc/N6S5-4PGN] find
  56. Grant-Chapman, H., Laird, E., & Venzke, C. (2021). Student Activity Monitoring Software Research Insights and Recommendations. Center for Democracy & Technology. [perma.cc/FY8G-WC2P] find
  57. Gröndahl, T., Pajola, L., Juuti, M., Conti, M., & Asokan, N. (2018). All You Need is “Love”: Evading Hate Speech Detection. Proceedings of the 11th ACM Workshop on Artificial Intelligence and Security, 2–12. [perma.cc/ T6P5-FRX5] find
  58. Hao, K. (2022, April 22). A new vision of artificial intelligence for the people. MIT Technology Review. [perma. cc/54U3-KU5C] find
  59. Heikkilä, M. (2022, November 14). We’re getting a better idea of AI’s true carbon footprint. MIT Technology Review. [perma.cc/8PWZ-ESJK] find
  60. Hutchinson, B., Prabhakaran, V., Denton, E., Webster, K., Zhong, Y., & Denuyl, S. (2020). Social Biases in NLP Models as Barriers for Persons with Disabilities. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5491–5501. [perma.cc/8FGR-P3FA]
  61. Izsak, P., Berchansky, M., & Levy, O. (2021). How to Train BERT with an Academic Budget. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 10644–10652. [perma.cc/8MPG-W2QE] find
  62. Joshi, P., Santy, S., Budhiraja, A., Bali, K., & Choudhury, M. (2020). The State and Fate of Linguistic Diversity and Inclusion in the NLP World. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 6282–6293. [perma.cc/82HQ-EH65]
  63. Kaack, L. H., Donti, P. L., Strubell, E., Kamiya, G., Creutzig, F., & Rolnick, D. (2022). Aligning artificial intelligence with climate change mitigation. Nature Climate Change, 12(6), Article 6. [perma.cc/7C4S-X2LH] find
  64. Khan, M., & Hanna, A. (2023). The Subjects and Stages of AI Dataset Development: A Framework for Dataset Accountability. Ohio State Technology Law Journal, 19. [perma.cc/XLG3-AP2J] find
  65. Kinchin, N., & Mougouei, D. (2022). What Can Artificial Intelligence Do for Refugee Status Determination? A Proposal for Removing Subjective Fear. International Journal of Refugee Law. [perma.cc/3KER-DZ5R] find
  66. Koehn, P., & Knowles, R. (2017). Six Challenges for Neural Machine Translation. Proceedings of the First Workshop on Neural Machine Translation, 28–39. [perma.cc/9WSQ-HQJY] find
  67. Kornai, A. (2013). Digital Language Death. PLOS ONE, 8(10), e77056. [perma.cc/MMZ8-C9VH] find
  68. Kreutzer, J., Caswell, I., Wang, L., Wahab, A., van Esch, D., Ulzii-Orshikh, N., Tapo, A., Subramani, N., Sokolov, A., Sikasote, C., Setyawan, M., Sarin, S., Samb, S., Sagot, B., Rivera, C., Rios, A., Papadimitriou, I., Osei, S., Suarez, P. O., … Adeyemi, M. (2022). Quality at a Glance: An Audit of…
  69. Kublik, V. (n.d.). EU/US Copyright Law and Implications on ML Training Data. Valohai. [perma.cc/LD3Z-RVW7] find
  70. Kupfer, M., & Muyumba, J. (2022). Language & Coloniality: Non-Dominant Languages in the Digital Landscape. Pollicy. [perma.cc/PM8N-Y9YW] find
  71. Laurençon, H., Saulnier, L., Wang, T., Akiki, C., Moral, A. V. del, Scao, T. L., Werra, L. V., Mou, C., Ponferrada, E. G., Nguyen, H., Frohberg, J., Šaško, M., Lhoest, Q., McMillan-Major, A., Dupont, G., Biderman, S., Rogers, A., Allal, L. B., Toni, F. D., … Jernite, Y. (2022, October 31). The BigSc…
  72. Lauscher, A., Ravishankar, V., Vulić, I., & Glavaš, G. (2020). From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 4483–4499. [perma.cc/ZJ3R-95JM]
  73. Lees, A., Sorensen, J., & Kivlichan, I. (2020). Jigsaw @ AMI and HaSpeeDe2: Fine-Tuning a Pre-Trained Comment-Domain BERT Model. In V. Basile, D. Croce, M. Maro, & L. C. Passaro (Eds.), EVALITA Evaluation of NLP and Speech Tools for Italian—December 17th, 2020 (pp. 40–47). Accademia University Press… find
  74. Lees, A., Tran, V. Q., Tay, Y., Sorensen, J., Gupta, J., Metzler, D., & Vasserman, L. (2022). A New Generation of Perspective API: Efficient Multilingual Character-level Transformers. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 3197–3207. [perma.cc/5K82-WG8J… find
  75. Libovický, J., Rosa, R., & Fraser, A. (2019). How Language-Neutral is Multilingual BERT? (arXiv:1911.03310). arXiv. [perma.cc/96RW-WXBL] — [arXiv:1911.03310](https://arxiv.org/abs/1911.03310) find
  76. Lin, X. V., Mihaylov, T., Artetxe, M., Wang, T., Chen, S., Simig, D., Ott, M., Goyal, N., Bhosale, S., Du, J., Pasunuru, R., Shleifer, S., Koura, P. S., Chaudhary, V., O’Horo, B., Wang, J., Zettlemoyer, L., Kozareva, Z., Diab, M., … Li, X. (2022). Few-shot Learning with Multilingual Generative Langu…
  77. Lokhov, I. (2021, January 28). Why are there so many Wikipedia articles in Swedish and Cebuano? Datawrapper Blog. [perma.cc/WDL2-TF53] find
  78. Luccioni, A., & Viviano, J. (2021). What’s in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Pa… find
  79. Lunden, I. (2023, March 14). Nabla, a digital health startup, launches Copilot, using GPT-3 to turn patient conversations into action. TechCrunch. [perma.cc/MK55-SV54] find
  80. Martin, G., Mswahili, M. E., Jeong, Y.-S., & Woo, J. (2022). SwahBERT: Language Model of Swahili. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 303–313. [perma.cc/3ZP6-V6AJ] find
  81. Martin, L., Muller, B., Suárez, P. J. O., Dupont, Y., Romary, L., de la Clergerie, É. V., Seddah, D., & Sagot, B. (2020). CamemBERT: A Tasty French Language Model. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 7203–7219. [perma.cc/76EU-4LTM]
  82. Masakhane. (n.d.). Masakhane. Retrieved December 21, 2022. [perma.cc/A7SA-ALPM] find
  83. Maundu, C. (2023, February 21). How language denies people access to public information. Nation. [perma. cc/8C4B-JS3Y] find
  84. Mazzucato, M. (2014). The entrepreneurial state: Debunking public vs. private sector myths (Revised edition). Anthem Press. find
  85. Meta AI. (2019, November 7). XLM-R: State-of-the-art cross-lingual understanding through self-supervision. Meta AI. [perma.cc/J55N-4MV5] find
  86. Micallef, K., Gatt, A., Tanti, M., van der Plas, L., & Borg, C. (2022). Pre-training Data Quality and Quantity for a Low-Resource Language: New Corpus and BERT Models for Maltese. Proceedings of the Third Workshop on Deep Learning for Low-Resource Natural Language Processing, 90–101. [perma.cc/QY8V-…
  87. Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013, September 6). Efficient Estimation of Word Representations in Vector Space. International Conference on Learning Representations. [perma.cc/T869-PDX4]
  88. Muller, B., Anastasopoulos, A., Sagot, B., & Seddah, D. (2021). When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu…
  89. Nadkarni, P. M., Ohno-Machado, L., & Chapman, W. W. (2011). Natural language processing: An introduction. Journal of the American Medical Informatics Association : JAMIA, 18(5), 544–551. [perma.cc/72PK-UGK9] find
  90. Nekoto, W., Marivate, V., Matsila, T., Fasubaa, T., Fagbohungbe, T., Akinola, S. O., Muhammad, S., Kabongo Kabenamualu, S., Osei, S., Sackey, F., Niyongabo, R. A., Macharm, R., Ogayo, P., Ahia, O., Berhe, M. M., Adeyemi, M., Mokgesi-Selinga, M., Okegbemi, L., Martinus, L., … Bashir, A. (2020). Parti…
  91. Nguer, E. M., Lo, A., Dione, C. M. B., Ba, S. O., & Lo, M. (2020). SENCORPUS: A French-Wolof Parallel Corpus. Proceedings of the Twelfth Language Resources and Evaluation Conference, 2803–2811. [perma.cc/ NBE7-QCZW] find
  92. Nguyen, T. (2020, November 27). Why fake news is so hard to combat in Asian American communities. Vox. [perma.cc/45GF-UUEC] find
  93. Nicholas, G. (2020). Explaining Algorithmic Decisions. Georgetown Law Technology Review, 4(711), 20. [perma. cc/UD7D-HF6F] find
  94. Nicholas, G. (2022). Shedding Light on Shadowbanning. Center for Democracy & Technology. [perma.cc/D2TS-Y92D] find
  95. Nkemelu, D., Shah, H., Essa, I., & Best, M. L. (2023). Tackling Hate Speech in Low-resource Languages with Context Experts. International Conference on Information & Communication Technologies and Development, Washington, USA. [perma.cc/5QK7-GTMR] find
  96. NLLB Team, Costa-jussà, M. R., Cross, J., Çelebi, O., Elbayad, M., Heafield, K., Heffernan, K., Kalbassi, E., Lam, J., Licht, D., Maillard, J., Sun, A., Wang, S., Wenzek, G., Youngblood, A., Akula, B., Barrault, L., Gonzalez, G. M., Hansanti, P., … Wang, J. (2022). No Language Left Behind: Scaling H… find
  97. Okerlund, J., Klasky, E., Middha, A., Kim, S., Rosenfeld, H., Kleinman, M., & Parthasarathy, S. (2022). What’s in the Chatterbox? Large Language Models, Why They Matter, and What We Should Do About Them. University of Michigan. [perma.cc/8SXE-RSYE] find
  98. OpenAI. (2023). GPT-4 Technical Report (arXiv:2303.08774). arXiv. [perma.cc/6ACB-LZYC]
  99. Orife, I., Kreutzer, J., Sibanda, B., Whitenack, D., Siminyu, K., Martinus, L., Ali, J. T., Abbott, J., Marivate, V., Kabongo, S., Meressa, M., Murhabazi, E., Ahia, O., van Biljon, E., Ramkilowan, A., Akinfaderin, A., Öktem, A., Akin, W., Kioko, G., … Bashir, A. (2020). Masakhane—Machine Translation… — [arXiv:2003.11529](https://arxiv.org/abs/2003.11529) find
  100. Patel, F., Levinson-Waldman, R., Koreh, R., & DenUyl, S. (2020). Social Media Monitoring. Brennan Center for Justice. [perma.cc/N5LF-ZKP2] find
  101. Phillipson, R. (1992). Linguistic Imperialism. Oxford University Press. find
  102. Pires, T., Schlinger, E., & Garrette, D. (2019). How Multilingual is Multilingual BERT? Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 4996–5001. [perma.cc/4DPF-LWWX]
  103. Png, M.-T. (2022). At the Tensions of South and North: Critical Roles of Global South Stakeholders in AI Governance. 2022 ACM Conference on Fairness, Accountability, and Transparency, 1434–1445. [perma.cc/ Z7HD-3T4A] find
  104. Polignano, M., Basile, P., Degemmis, M., Semeraro, G., & Basile, V. (2019). AlBERTo: Italian BERT Language Understanding Model for NLP Challenging Tasks Based on Tweets. Sixth Italian Conference on Computational Linguistics, Bari, Italy. [perma.cc/RBY9-4JHJ]
  105. Prabhakaran, V., Mitchell, M., Gebru, T., & Gabriel, I. (2022). A Human Rights-Based Approach to Responsible AI (arXiv:2210.02667). arXiv. [perma.cc/R97H-WQSK] find
  106. Prates, M., Avelar, P., & Lamb, L. (2020). Assessing gender bias in machine translation: A case study with Google Translate. Neural Computing and Applications, 32. [perma.cc/CGK2-NMU2]
  107. Pym, A., Ayvazyan, N., & Prioleau, J. M. (2022). Should raw machine translation be used for public-health information? Suggestions for a multilingual communication policy in Catalonia. Just. Journal of Language Rights & Minorities, Revista de Drets Lingüístics i Minories, 1(1–2), 71–99. [perma.cc/HS… find
  108. Raji, D., Denton, E., Bender, E. M., Hanna, A., & Paullada, A. (2021). AI and the Everything in the Whole Wide World Benchmark. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 1. [perma.cc/EX84-X9BQ] find
  109. Razeghi, Y., Logan IV, R. L., Gardner, M., & Singh, S. (2022). Impact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning. Findings of the Association for Computational Linguistics: EMNLP 2022, 840–854. [perma.cc/SMG9-BSKV]
  110. Reid, M., & Artetxe, M. (2022). On the Role of Parallel Data in Cross-lingual Transfer Learning (arXiv:2212.10173). arXiv. [perma.cc/83GW-CVXX] find
  111. Richter, F. (n.d.). English Is the Internet’s Universal Language. Statista Infographics. Retrieved December 14, 2022, from [perma.cc/WW7B-7X37] find
  112. Rionda, V. P. S., & Mejia, J. C. U. (2021). PretorIA y la automatización del procesamiento de causas de derechos humanos. Derechos Digitales and Dejusticia. [perma.cc/65MQ-X484] find
  113. Rowe, J. (2022, March 2). Marginalised languages and the content moderation challenge. Global Partners Digital. [perma.cc/GU4K-5HBE] find
  114. Ruder, S., Constant, N., Botha, J., Siddhant, A., Firat, O., Fu, J., Liu, P., Hu, J., Garrette, D., Neubig, G., & Johnson, M. (2021). XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 10215–…
  115. Santa Clara Principles. (2021). Santa Clara Principles on Transparency and Accountability in Content Moderation. Santa Clara Principles. [perma.cc/T623-AVW6] find
  116. Santy, S., Kummerfeld, J., & Rubio, H. (2023). Languages mentioned in Paper Abstracts. ACL Rolling Review. [perma.cc/EQU9-5CWQ] find
  117. Savoldi, B., Gaido, M., Bentivogli, L., Negri, M., & Turchi, M. (2021). Gender Bias in Machine Translation. Transactions of the Association for Computational Linguistics, 9, 845–874. [perma.cc/9K3F-5VBZ]
  118. Schwenk, H. (2019, January 22). LASER natural language processing toolkit—Engineering at Meta. Meta AI. [perma.cc/46JG-AZ4T] find
  119. Sengupta, P. B., Claudia Pozo, Anasuya. (2022, March 31). Does the internet speak your language? Launching the first-ever State of the Internet’s Languages report. Whose Knowledge? [https://perma.cc/9KCX-M863] find
  120. Sharir, O., Peleg, B., & Shoham, Y. (2020). The Cost of Training NLP Models: A Concise Overview (arXiv:2004.08900). arXiv. [perma.cc/8KVV-C6P2] — [arXiv:2004.08900](https://arxiv.org/abs/2004.08900) find
  121. Shenkman, C., Thakur, D., & Llansó, E. (2021). Do You See What I See? Capabilities and Limits of Automated Multimedia Content Analysis. Center for Democracy & Technology. [perma.cc/W85T-HQQF] find
  122. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., Kluska, A., Lewkowycz, A., Agarwal, A., Power, A., Ray, A., Warstadt, A., Kocurek, A. W., Safaya, A., Tazarv, A., … Wu, Z. (2022). Beyond the Imitation Game: Quantify…
  123. Stadthagen-Gonzalez, H., Imbault, C., Pérez Sánchez, M. A., & Brysbaert, M. (2017). Norms of valence and arousal for 14,031 Spanish words. Behavior Research Methods, 49(1), 111–123. [perma.cc/7FWX-Z3JD] find
  124. Strubell, E., Ganesh, A., & McCallum, A. (2019). Energy and Policy Considerations for Deep Learning in NLP. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 3645–3650. [perma.cc/9P4Y-J4HT]
  125. Talat, Z., Névéol, A., Biderman, S., Clinciu, M., Dey, M., Longpre, S., Luccioni, S., Masoud, M., Mitchell, M., Radev, D., Sharma, S., Subramonian, A., Tae, J., Tan, S., Tunuguntla, D., & Wal, O. van der. (2022). You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings…
  126. Tay, Y., Tran, V. Q., Ruder, S., Gupta, J., Chung, H. W., Bahri, D., Qin, Z., Baumgartner, S., Yu, C., & Metzler, D. (2022, February 23). Charformer: Fast Character Transformers via Gradient-based Subword Tokenization. International Conference on Learning Representations 2022. [perma.cc/YRL4-E7DT]
  127. Teich, E. (2003). Cross-Linguistic Variation in System and Text: A Methodology for the Investigation of Translations and Comparable Texts. In Cross-Linguistic Variation in System and Text. De Gruyter Mouton. [perma.cc/L8A8-RH8B] find
  128. Torbati, Y. (2019, September 26). Google Says Google Translate Can’t Replace Human Translators. Immigration Officials Have Used It to Vet Refugees. ProPublica. [perma.cc/ZUN6-LHA5] find
  129. Toyama, K. (2015). Geek heresy: Rescuing social change from the cult of technology. PublicAffairs. find
  130. Tsvetkov, Y., Sitaram, S., Faruqui, M., Lample, G., Littell, P., Mortensen, D., Black, A. W., Levin, L., & Dyer, C. (2016). Polyglot Neural Language Models: A Case Study in Cross-Lingual Phonetic Representation Learning. Proceedings of the 2016 Conference of the North American Chapter of the Associa… find
  131. United Nations Human Rights Office of the High Commissioner. (n.d.). ​​ Economic, social and cultural rights. OHCHR. [perma.cc/Y6MK-SZZ4] Vallee, H. Q. la, & Duarte, N. (2019). Algorithmic Systems in Education: Incorporating Equity and Fairness When Using Student Data. Center for Democracy and Techn… find
  132. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017, December 5). Attention Is All You Need. Advances in Neural Information Processing Systems. [perma. cc/2ZDX-Z796]
  133. Vincent, J. (2022, November 2). Google plans giant AI language model supporting world’s 1,000 most spoken languages. The Verge. [perma.cc/3Y48-X7WV] find
  134. Vincent, J. (2023a, January 17). Getty Images is suing the creators of AI art tool Stable Diffusion for scraping its content. The Verge. [perma.cc/4CXS-3WNN] find
  135. Vincent, J. (2023b, March 15). OpenAI co-founder on company’s past approach to openly sharing research: “We were wrong.” The Verge. [perma.cc/DPL6-4PD2] find
  136. Vitulli, M. A. (2018). Writing Women in Mathematics Into Wikipedia. Notices of the American Mathematical Society, 65(03), 330–334. [perma.cc/X73F-AZPM] find
  137. Volansky, V., Ordan, N., & Wintner, S. (2015). On the features of translationese. Digital Scholarship in the Humanities, 30(1), 98–118. [perma.cc/7F8S-3YXK] find
  138. Wang, C., Cho, K., & Gu, J. (2020). Neural Machine Translation with Byte-Level Subwords. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05), Article 05. [perma.cc/5DL7-XSSP]
  139. Wang, Z., K, K., Mayhew, S., & Roth, D. (2020). Extending Multilingual BERT to Low-Resource Languages. Findings of the Association for Computational Linguistics: EMNLP 2020, 2649–2656. [perma.cc/ZNC8-C9E7] find
  140. Williams, A., Miceli, M., & Gebru, T. (2022). The Exploited Labor Behind Artificial Intelligence. Noēma. [perma. cc/GE8H-7SUN] find
  141. Wu, S., & Dredze, M. (2019). Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 833–844. [perma.cc/EJ3…
  142. Wu, S., & Dredze, M. (2020). Are All Languages Created Equal in Multilingual BERT? Proceedings of the 5th Workshop on Representation Learning for NLP, 120–130. [perma.cc/5E6X-NNAA]
  143. Yu, S., Sun, Q., Zhang, H., & Jiang, J. (2022). Translate-Train Embracing Translationese Artifacts. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 362– 370. [perma.cc/7F8C-EYM6] find
  144. Zhang, S., Frey, B., & Bansal, M. (2022, April 25). How can NLP Help Revitalize Endangered Languages? A Case Study and Roadmap for the Cherokee Language. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). find
  145. ACL 2022, Dublin, Ireland. [perma.cc/2XF2-2GDC] find
  146. Zhang, Y., Warstadt, A., Li, X., & Bowman, S. R. (2021). When Do You Need Billions of Words of Pretraining Data? Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers),… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nicholas-2023-lost-translation-large-language-models/