📄 Discriminating Non-Native English with 350 Words

Summary

The MITRE team took part in the BEA-8 shared task on native language identification (NLI). The goal was to guess which of eleven first languages an author speaks, using only one English TOEFL essay of roughly 350 words. Their system uses no hand-written rules. It combines three independently trained eleven-way classifiers:

• Carnie, an in-house multiclass Winnow2 learner over hashed binary word and character n-grams;
• SRILM, with one trigram language model per language;
• LIBLINEAR, L2-regularized logistic regression.

A naive Bayes layer merges their predicted labels and discretized confidence scores, with its parameters estimated on the development set. The best ensemble reached 82.6% accuracy, which is statistically tied with the top systems among 29 teams. It also ranked first on Arabic by F-measure. The paper's broader lesson is that shallow lexical evidence carries a lot of signal about an author's L1, especially when diverse learners with uncorrelated errors are combined. The authors also caution that topic may leak into such lexical cues.

Key points

• Data: the TOEFL corpus from ETS has 12,100 essays covering 11 L1s. Each language has 1000 training, 100 development and 100 test essays, so chance accuracy is about 9%.
• Features: binary, case-sensitive word and character n-grams plus surface statistics, hashed with MurmurHash3. Each feature type has its own namespace. POS n-grams up to length 5 come from the Stanford tagger with Penn Treebank tags.
• Final ensemble: naive Bayes over Carnie, SRILM (default trigram, Good-Turing, open vocabulary, each essay treated as one sentence) and LIBLINEAR (dual solver). A grid search chose λ = 0.03125 and two score partitions.
• Binary versus counted features: binary presence features consistently beat frequency-weighted ones. For the JCarafe max-ent model the switch alone gave about 10% more accuracy. The proficiency rating by itself predicted L1 at only 14.9%.
• Out-of-domain data: on the Japanese/Korean pair, naively adding outside data lowered accuracy from 86.5% to 82.0%. EasyAdapt feature duplication raised it to 88.5%.

Interesting ideas

• I1. Confusion-aware voting: each component's vote is converted into a probability of the true label using confusion counts from held-out data, and a coarse score bin is folded in. This lets weak, differently-biased models be combined with very little tuning data.
• I2. EasyAdapt duplication: every feature gets a general copy and a source-specific copy, so the learner can separate shared patterns from domain-specific ones. This turned harmful extra data into a gain.
• I3. Feature tracks: each feature type gets its own namespace, so identical strings such as the word "i" and the character "i" never collide after hashing. This is a cheap way to keep many feature families together in one sparse model.
• I4. Trivial baselines first: cheap predictors such as word length, proficiency level and vowel/consonant cluster distributions were measured before building anything serious. This shows how much signal sits at each surface level and sets a floor that real models must beat.
• I5. POS n-grams for learner errors: POS n-gram features exposed systematic non-native constructions, such as a verb left in its bare, unconjugated form right after a personal pronoun. Some fully grammatical patterns also turned out to be typical of one L1.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): witness parsers voting on silver arcs or labels could be weighted by their per-label confusion tables and confidence bins, instead of using plain majority agreement.
• I2 (tree): when the perceptron parser trains on mixed treebanks or converted UD dialects, per-source feature copies could stop one dialect's conventions from polluting the others.
• I3 (tokens): tagger and morphology features drawn from words, suffixes and character windows could be namespaced the same way, keeping their weights distinct and inspectable.
• I4 (learn): such shallow predictors make natural negative controls, so a gain is only credited when it clearly beats what length or surface statistics alone can explain.
• I5 (tree): tag-sequence patterns like these could flag likely learner errors before parsing, so the parser or repair rules handle non-native input explicitly rather than misattaching it.