Elamur Research⌕
☰
home / papers / 2022 / Should We Ban English NLP for a Year?

Should We Ban English NLP for a Year?

Paper · 2022 · Anders Søgaard

Summary

This position paper argues that NLP widens global inequality. It does so in two ways: by concentrating on English, and by building systems around young, urban, tech-savvy users. Søgaard reviews estimates that about two thirds of research at top venues covers only English, with no improvement over a decade. He then runs a small simulation in which user uptake depends on how well a model serves each group. Under that assumption, an initially small group falls further behind over time instead of catching up.

He sorts common defences of this imbalance using Bank's (2018) five types: Opportunity, Desert, Procedure, Need and Reference. He rejects all of them except special Need. He also challenges the claim that data scarcity is to blame, since GPT-2-scale text exists for about the hundred largest languages.

As remedies, he borrows three climate-policy tools: cap-and-trade, a carbon tax and car-free Sundays. He favours the last, which would mean a temporary ban, such as desk-rejecting English-only papers at ACL 2023. He argues the field adapts quickly, so any cost would be short-lived.

Key points

  • Estimates of English-only work at top NLP venues range from 50% to 90%, averaging about two thirds, and the share has not fallen over ten years.
  • The simulation setup:
    • Model: a depth-4 binary decision tree trained on scikit-learn "circles" data (four datasets).
    • Start: 55 users, of whom only 5 belong to the minority group.
    • Growth: up to 20 users join per step, in proportion to how well the model serves each group.
    • Result: the accuracy gap grows over time.
  • The data-scarcity argument is weak:
    • GPT-2 was trained on 40GB of text, and comparable amounts exist for the top-100 languages.
    • w3techs counts over two million web pages for each of these languages.
    • Labelled resources such as UD (Nivre et al., 2020) also exist for many languages.
  • "State of the art" lists on the ACL Wiki, for example POS tagging, are ranked by English Penn Treebank scores. Papers on other languages also receive fewer citations.
  • The proposed ban would have redirected 702 papers, the number ACL 2022 accepted. The author admits it could be gamed with machine-translated datasets.

Interesting ideas

  • I1. If a system learns mostly from the users or data it already handles well, its weak areas become relatively weaker with each round; a toy simulation is a cheap way to expose this loop.
  • I2. Defining "best" by one benchmark (Penn Treebank) quietly sets research priorities, so evaluation choices act as policy.
  • I3. Of the common reasons for unequal treatment, only genuine special Need holds up; convenience and precedent do not.
  • I4. Machine-translated datasets carry known biases, so native data should be preferred whenever it exists.
  • I5. Systems tuned to early-adopter speakers underperform for other demographics, so coverage of underserved users deserves its own measurement.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Track per-genre and per-construction accuracy across silver-data and self-training rounds, so that witness-parser agreement does not keep enriching only the easy sentences.
  • I2 (tree): Report parser and gparse quality across several UD English treebanks and registers, not just one headline LAS, so that progress is not tuned to a single corpus.
  • I3 (tokens): Allow hand-written special cases in the tokenizer and morphology only where a phenomenon truly requires them, and record that justification next to the rule.
  • I4 (back): When testing translation through the meta-world, use natively written texts in the target language rather than MT-derived sets, which would flatter the system.
  • I5 (answer): Add test items with non-native and non-standard English to the readers' evaluation, to check that answers do not degrade for such writers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CopenhagenCopenhagen · Denmark115 works · 15,292 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Anders Søgaard. 2022. Should We Ban English NLP for a Year?. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.emnlp-main.351

@inproceedings{sogaard2022should,
  title = {{Should We Ban English NLP for a Year?}},
  author = {Anders Søgaard},
  booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.351},
  url = {https://doi.org/10.18653/v1/2022.emnlp-main.351}
}

References

48 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Silvio Amir, Jan-Willem van de Meent, and Byron Wallace. 2021. On the impact of random seeds on the fairness of clinical classifiers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3808–3823, Online. Association for Computational Linguistics. find
  2. S. Ananiadou, John McNaught, and P. Thompson. 2012. The English Language in the Digital Age. META-NET White Paper Series. Springer Nature, United States. EC FP7 PSP Project METANET4U. ↗
  3. Julian Bank. 2018. Mr. Winterkorn’s pay: A typology of justification patterns of income inequality. Social Justice Research, 29:228–252. find
  4. Julia Black and Andrew Douglas Murray. 2019. Regulating ai and machine learning: Setting the regulatory agenda. Eur. J. Law Technol., 10. find
  5. Damian Blasi, Antonios Anastasopoulos, and Graham Neubig. 2022. Systematic inequalities in language technology performance across the world’s languages. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5486–5505, Dublin, Ireland. Association for Computational Linguistics.
  6. Ilias Chalkidis, Tommaso Pasini, Sheng Zhang, Letizia Tomada, Sebastian Schwemer, and Anders Søgaard. 2022. FairLex: A multilingual benchmark for evaluating fairness in legal text processing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4389–4406, Dublin, Ireland. Association for Computational Linguistics. find
  7. Kawin Ethayarajh and Dan Jurafsky. 2020. Utility is in the eye of the user: A critique of NLP leaderboards. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4846–4853, Online. Association for Computational Linguistics. find
  8. Sina Fazelpour and Zachary C. Lipton. 2020. Algorithmic fairness from a non-ideal perspective. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, AIES ’20, page 57–63, New York, NY, USA. Association for Computing Machinery. find
  9. Sina Fazelpour, Zachary C. Lipton, and David Danks. 2022. Algorithmic fairness and the situated dynamics of justice. Canadian Journal of Philosophy, 52(1):44– 60. ↗
  10. Siyuan Feng, Olya Kudina, Bence Mark Halpern, and Odette Scharenborg. 2021. Quantifying bias in automatic speech recognition. CoRR, abs/2103.15122. ↗
  11. Svitlana Galeshchuk, Ju Qiu, and Julien Jourdan. 2019. Sentiment analysis for multilingual corpora. In Proceedings of the 7th Workshop on Balto-Slavic Natural Language Processing, pages 120–125, Florence, Italy. Association for Computational Linguistics. find
  12. Andrew Glazener, James Wylie, Willem van Waas, and Haneen Khreis. 2022. The impacts of car-free days and events on the environment and human health. Current Environmental Health Reports, 9(2). Tahmid Hasan, Abhik Bhattacharjee, Md. ↗
  13. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. XL-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4693–4703, Online. Association for Computational Linguistics. find
  14. Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. 2022. Challenges and strategies in cross-cultural NLP. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6997–7013, Dublin, Ireland. Association for Computational Linguistics. ↗
  15. Sverker C. Jagers, Erick Lachapelle, Johan Martinsson, and Simon Matti. 2021. Bridging the ideological gap? how fairness perceptions mediate the effect of revenue recycling on public support for carbon taxes in the united states, canada and germany. Review of Policy Research, 38(5):529–554. ↗
  16. Sébastien Jean, Orhan Firat, and Melvin Johnson. 2019. Adaptive scheduling for multi-task learning. In ArXiv 1909.06434. ↗
  17. Zhijing Jin, Geeticka Chauhan, Brian Tse, Mrinmaya Sachan, and Rada Mihalcea. 2021. How good is NLP? a sober look at NLP tasks through the lens of social impact. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3099–3113, Online. Association for Computational Linguistics. ↗
  18. Brian Larson. 2017. Gender as a variable in natural-language processing: Ethical considerations. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing, pages 1–11, Valencia, Spain. Association for Computational Linguistics. find
  19. Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: Evaluating cross-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7315– 7330, Online. Association for Computational Linguistics. ↗
  20. Miryam de Lhoneux, Sheng Zhang, and Anders Søgaard. 2022. Zero-shot dependency parsing with worst-case aware automated curriculum learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 578–587, Dublin, Ireland. Association for Computational Linguistics. find
  21. Mike Li, Hongseok Namkoong, and Shangzhou Xia. 2021. Evaluating model performance under worst-case subpopulations. In Advances in Neural Information Processing Systems, volume 34, pages 17325– 17334, Vancouver, CA. Curran Associates, Inc. Ting-An Lin and Po-Hsuan Cameron Chen. forthcoming. Artificial intelligence in a structurally unjust society. Feminist Philosophy Quarterly. find
  22. Shengchao Liu, Yingyu Liang, and Anthony Gitter. 2019. Loss-balanced task weighting to reduce negative transfer in multi-task learning. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):9977–9978. find
  23. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330. ↗
  24. Nina Markl. 2022. Language variation and algorithmic bias: Understanding algorithmic bias in british english automatic speech recognition. In 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’22, page 521–534, New York, NY, USA. Association for Computing Machinery. find
  25. Gustavo Marrero. 2021. Does Race and Gender Inequality Impact Income Growth? The World Bank, Washington, D.C. ↗
  26. Ralf Martin, Laure B. de Preux, and Ulrich J. Wagner. 2014. The impact of a carbon tax on manufacturing: Evidence from microdata. Journal of Public Economics, 117:1–14. ↗
  27. Tridas Mukhopadhyay and Vandana Mangal. 1997. Direct and indirect impacts of information technology applications on productivity: A field study. International Journal of Electronic Commerce, 1(3):83–102. ↗
  28. Kai Nielsen. 1979. Radical egalitarian justice: Justice as equality. Social Theory and Practice, 5(2):209–226. find
  29. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4034–4043, Marseille, France. European Language Resources Association. ↗
  30. Emily Öhman, Marc Pàmies, Kaisla Kajava, and Jörg Tiedemann. 2020. XED: A multilingual dataset for sentiment analysis and emotion detection. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6542–6552, Barcelona, Spain (Online). International Committee on Computational Linguistics. ↗
  31. Gustavo Paetzold and Lucia Specia. 2016. Understanding the lexical simplification needs of non-native speakers of English. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 717– 727, Osaka, Japan.
  32. The COLING 2016 Organizing Committee. find
  33. Janet Peace and Robert N. Stavins. 2010. Meaningful and Cost Effective Climate Policy: The Case for Cap and Trade. Pew Center on Global Climate Change, Arlington, VA. F-28. find
  34. Edoardo Maria Ponti, Rahul Aralikatte, Disha Shrivastava, Siva Reddy, and Anders Søgaard. 2021. Minimax and neyman–Pearson meta-learning for outlier languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1245– 1260, Online. Association for Computational Linguistics. find
  35. John Rawls. 1971. A Theory of Justice, 1 edition. Belknap Press of Harvard University Press, Cambridge, Massachussets. find
  36. Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2022. Square one bias in NLP: Towards a multidimensional exploration of the research manifold. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2340–2354, Dublin, Ireland. Association for Computational Linguistics. find
  37. Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. 2020. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. ↗
  38. Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2020. MLSUM: The multilingual summarization corpus. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8051–8067, Online. Association for Computational Linguistics.
  39. Jasdeep Singh, Bryan McCann, Richard Socher, and Caiming Xiong. 2019. BERT is not an interlingua and the bias of tokenization. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 47–55, Hong Kong, China. Association for Computational Linguistics. find
  40. Masao Utiyama and Hitoshi Isahara. 2007. A comparison of pivot methods for phrase-based statistical machine translation. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages 484–491, Rochester, New York. Association for Computational Linguistics.
  41. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. In Advances in Neural Information Processing Systems, volume 33, pages 12388–12401, Vancouver, CA. Curran Associates, Inc. find
  42. Andy Way, Georg Rehm, Jane Dunne, Maria Giagkou, José Manuel Gomez-Perez, Jan Hajič, Stefanie Hegele, Martin Kaltenböck, Teresa Lynn, Katrin Marheinecke, Natalia Resende, Inguna Skadin , a amd Marcin Skowron, Tea Vojtěchová, and Annika Grützner-Zahn. 2022. Report on the state of language technology in 2030. Available at https://european-language-equality.eu/wp-content/ uploads/2022/05/ELE___Deliverable_D2_18_ _Report_on_State_of_LT_in_2030_.pdf. ↗
  43. Stefan Wörner and Thomas Reiss. 2001. The direct and indirect impacts of new technologies on employment: The example of the German biotechnology sector. Science and Public Policy, 28(5):371–380. ↗
  44. Victoria Yaneva, Constantin Orasan, Le An Ha, and Natalia Ponomareva. 2019. A survey of the perceived text adaptation needs of adults with autism. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 1356–1363, Varna, Bulgaria. INCOMA Ltd. find
  45. Sheng Zhang, Xin Zhang, Weiming Zhang, and Anders Søgaard. 2021. Sociolectal analysis of pretrained language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4581–4588, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  46. Sheng Zhang, Xin Zhang, Weiming Zhang, and Anders Søgaard. 2020. Worst-case-aware curriculum learning for zero and few shot transfer. ↗
  47. T. V. Smith. 1935. Opportunism. The International Journal of Ethics, 45(2):235–239. find
  48. Asa Cooper Stickland and Iain Murray. 2019. BERT and PALs: Projected attention layers for efficient adaptation in multi-task learning. In ICML. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sogaard-2022-should-we-ban-english-nlp/