📄 Detecting English Writing Styles For Non-native Speakers

Summary

The paper asks whether an author's native language can be detected from their informal English. It also asks whether those signals reveal how languages relate to one another. Earlier studies relied on learner essays such as ICLE, which cover few languages and topics. Al-Rfou' instead mined signed comments from English Wikipedia talk pages and took each author's native language and English fluency from the Babel boxes users add to their profiles. After filtering, the dataset has 9857 users and 589,228 comments. Each comment is scored against per-class n-gram models built over words, characters and POS tags. These scores, plus stop-word frequencies and simple length statistics, feed logistic regression and linear SVM classifiers. Separating native from non-native writers reaches about 74.5% accuracy. Six-way native-language and five-way language-family tasks reach about 50%. Clustering the classifier's confusion patterns recovers groupings close to known language families. This suggests that L1 influence leaves a measurable mark even on short, colloquial, topically varied text.

Key points

• Data source: English Wikipedia talk-page comments, with labels taken from Babel boxes.
  • About 60K users declared language skills, and 47% of them were English natives.
  • About 12M comments were found, of which 2.4M come from users with known skills.
• Filtering:
  • Only the 20 most frequent native languages are kept.
  • English natives are limited to US English, and users claiming several native languages are dropped.
  • Each comment must contain a minimum of 20 tokens.
  • Every proper noun is swapped for its POS tag, and any non-ASCII character becomes a placeholder.
  • Classes are balanced, and the data is split 70/10/20 into train, dev and test.
• Features:
  • For each class there are 12 n-gram models: orders 1–4 over words, characters and POS tags.
  • A comment's score against a model is a sum of log2 counts (unseen grams count as 1), giving 72 features for six classes.
  • Extra features are relative frequencies of 125 NLTK stop words, average word length, comment length and sentence count.
• Accuracy:
  • Native vs non-native: 74.53% (SVM) and 74.45% (LR) on about 322K comments.
  • Six languages (US-EN, German, Spanish, French, Russian, Dutch): 50.27%. Russian was easiest to identify, and German vs Dutch was most confused.
  • Five language families: 50.81%.
  • Twenty languages: 25%.
• Further findings:
  • Error rises when the non-native class contains only high-fluency users (levels 3–5) rather than basic ones (0–2).
  • Learning curves climb from about 37% to over 50% as data grows, so more data would still help.
  • Affinity propagation on the 20-language confusion matrix yields clusters largely matching linguistic families.

Interesting ideas

• I1. Compact class scores: Instead of feeding raw sparse n-grams to the classifier, each class gets its own n-gram models, and a text is reduced to a few log-count match scores per class, which keeps the feature vector tiny and interpretable.
• I2. Confusion matrix as similarity: Treating a classifier's confusion matrix as a similarity matrix and clustering it reveals hidden structure among the classes, here language families.
• I3. Masking topic leaks: Proper nouns are replaced by POS tags and foreign characters by a placeholder, so the classifier cannot cheat on topic or script.
• I4. Graded labels as a sanity check: Self-reported fluency levels act as a dose-response test: if the labels are meaningful, more fluent groups should be harder to separate, and they are.
• I5. Reading top features as an error catalogue: Ranking n-grams by z-score per class turns the classifier into a list of concrete non-native habits. Examples are a space before commas, missing apostrophes, lowercase "earth", dropped or extra "the", and "l" written for "ll".

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Per-source n-gram scores (gold treebank vs. silver vs. noisy web) could serve as cheap features telling the trainer which register a sentence resembles, without a bag-of-words model.
• I2 (learn): Clustering the confusion between UD relation labels, or disagreements among witness parsers, could show which labels are systematically interchangeable and group self-induced repair rules accordingly.
• I3 (learn): The same masking can act as a negative control in evaluation, checking that an answerer or frame extractor still works when names are anonymised, which shows it is not memorising entities.
• I4 (learn): Silver data could be validated the same way, by checking that parser agreement degrades steadily as known noise levels increase rather than jumping randomly.
• I5 (tokens): The tokenizer and tagger could be tested against exactly these patterns (" ,", "dont", missing articles) so that non-native input still yields sound lemmas and tags downstream.