📄 Towards Better Inclusivity: A Diverse Tweet Corpus of English Varieties

Summary

Most English NLP data comes from the US and UK, so tools trained on it often fail on the many other Englishes used around the world. The authors build a Twitter corpus to help fix this at the data level instead of correcting models afterwards. They randomly sampled geo-located tweets from five non-western capitals (Accra, Islamabad, Manila, New Delhi, Singapore) and from New York and London as western reference points. Tweets with too few English words were filtered out. They then designed a six-category annotation scheme that places each tweet on a rough scale of distance from formal standard English, with code-switching as one of the categories. Multilingual NYU students, chosen for familiarity with the relevant varieties, annotated 3,564 tweets. As a demonstration, the authors ran three off-the-shelf language identifiers on the corpus. All three recognised non-western English as English far less often than American or British English. This matters because language ID is usually an early filter, so such errors silently drop speakers of these varieties from downstream pipelines.

Key points

• The raw collection has 170,800 tweets, 24,400 per city. Sampling took 100 random tweets per day from January 1 to September 1, 2022, to reduce topic effects.
• A threshold of at least 60% English words kept about 71.2% of tweets. Most removed tweets were non-English, but some non-English text with English-looking spellings survives.
• The annotation uses six labels along a pseudo-spectrum of standardness. Code-switched tweets count as English, since in a 100-tweet manual check 82% were at least half English.
• On 350 tweets per city, the western vs. non-western accuracy gaps were 21.82 (langid.py), 19.77 (spaCy-langdetect) and 32.58 (Google Translate API) points. Google labelled only 33.65% of Manila tweets as English.
• If code-switched tweets are not counted as English, the gaps shrink to 12.43, 7.33 and 8.93 points but remain in favour of western English.

Interesting ideas

• I1. The labels are mutually exclusive categories that are also ordered by how far a text sits from standard English, which supports both classification and graded analysis.
• I2. Location is used as a cheap proxy for dialect: tweets from a given capital are assumed more likely to show that country's variety, with no per-tweet dialect detection.
• I3. A simple tool's accuracy is reported per variety, under both a generous and a conservative definition of the correct answer. A bias that persists under both definitions is more convincing.
• I4. Early filtering steps such as language ID can quietly remove whole groups of users, which then biases everything downstream.
• I5. The authors sample a fixed number of random tweets per day across many months so that one news event cannot dominate the corpus vocabulary.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): Inputs could get a standardness grade before tagging, so the tokenizer and lemmatizer can switch on normalisation of abbreviations, contractions and phonetic spellings only for informal text.
• I2 (learn): Evaluation sets could be grouped by source region or genre, so that parser accuracy drops on particular varieties show up as separate, visible numbers instead of disappearing into an average.
• I3 (learn): Evaluations could report each score under a strict and a lenient gold standard, especially for informal or mixed-language input, so conclusions do not depend on one labelling choice.
• I4 (answer): Any reject-or-skip gate before parsing should log what it dropped and be audited on non-standard English samples, so that readers and QA do not silently refuse dialectal questions.
• I5 (learn): Silver-data harvesting could be spread evenly across time and sources, so that self-training on agreed parses does not overfit to a single topic or style.