📄 Detecting English Writing Styles For Non Native Speakers

Summary

The authors try to guess an English writer's native language from everyday, informal text at a much larger scale than earlier learner-corpus studies such as ICLE. They build the dataset from English Wikipedia. Users who fill in the voluntary Babel box supply the native-language label. Comments on article talk pages are linked to those users through their signatures, which are found with best-effort regular expressions. After filtering, 9857 users and 589,228 comments remain. For each class, they compute frequency profiles of word, character and part-of-speech 1–4-grams. Each comment is then scored against every profile with a sum of log counts. These scores, together with stop-word ratios and simple length statistics, are fed to logistic regression. Separating native from non-native writers reaches 74% accuracy. With six languages, accuracy is about 50%. The errors follow geographic and genealogical closeness: German and Dutch are the most often confused, and Asian writers are relatively easy to identify. This suggests that English writing style also reflects how close the writers' native languages are to each other.

Key points

• The data come from around 60 thousand Wikipedia users with Babel language boxes and roughly 12 million talk-page comments. Only 2.4 million of those comments could be tied to users with known skills. Final set after filtering: 9857 users, 589,228 comments, 19 popular native languages; for English, only EN-US natives are kept.
• Preprocessing: comments need at least 20 tokens, proper nouns are replaced by their tags, non-ASCII characters are masked, classes are balanced, and the split is 70/10/20.
• Features are per-class similarity scores for word, character and PoS n-grams with n=1–4, giving classes × 3 × 4 values. They are complemented by relative frequencies of 125 NLTK stop words, average word size and sentence count.
• Results:
  • native vs non-native: 74%;
  • six-way task (EN-US, German, Spanish, French, Russian, Dutch): best 50.275%;
  • learning curves rise from 37% to over 50% as more data is added.
• PoS 4-grams alone give 37.54%. Keeping only frequent 4-grams lowers accuracy (12.81% with the top 100). Roughly a third of comments contain unseen PoS 4-grams. Backing off to trigram/bigram/unigram reduced accuracy to about 30–34%.

Interesting ideas

• I1. Profiles that users declare about themselves, such as the Babel boxes, combined with signature-linked talk comments, turn a public wiki into a large labelled corpus without any annotation effort
• I2. Masking proper nouns and non-ASCII characters keeps the classifier from learning topic or script shortcuts in place of style
• I3. Each input is compressed into a handful of numbers: its similarity to class-level n-gram profiles over three views (words, characters, PoS tags) and four orders
• I4. To separate coverage failures from model failures, the authors evaluate separately on test items with no unseen n-grams for the correct class, where accuracy jumps to 78.41%
• I5. Rare PoS 4-grams carry real signal, since dropping them hurt badly, and the class with the widest coverage (US English) absorbed many errors

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Wikipedia metadata and signatures could cheaply supply author or register labels for evaluation sets, for example native vs non-native input to stress-test the parsers.
• I2 (learn): the same masking could act as a negative control, showing that a gain comes from syntax rather than from lexical topic leakage.
• I3 (tokens): per-sentence similarity to tag-sequence profiles of native and learner English could flag unusual input before tagging, and the flag would stay explainable.
• I4 (learn): reporting accuracy split by whether the input contains unseen patterns would show whether errors come from sparse data or from bad rules.
• I5 (tree): unseen tag 4-grams in the dependency input could be logged as possible learner constructions rather than discarded, and witness-parser agreement could be checked specifically on those cases.