📄 Interpretable Machine Learning for Societal Language Identification: Modeling English and German Influences on Portuguese Heritage Language

Summary

The paper asks whether the dominant language of the surrounding society leaves a measurable mark on how adolescents write in their heritage language. The authors call this task societal language identification (SLI). They studied European Portuguese texts written by heritage learners living in Germany and in the UK, covering CEFR levels B1, B2 and C1. To keep the classifier from relying on content cues, they used topic modeling to control for topic and also held proficiency constant. They then applied clustering-based undersampling so the two groups were balanced but still representative. Each text was described by automatically extracted complexity measures: lexical, morphological, syntactic, discursive, grammatical and count-based. An explainable boosting machine (EBM) learned to separate English-context texts from German-context texts. Morphological and lexical measures carried the most weight. Several of the distinguishing patterns resemble structures of the societal language, such as more passives in the English context. The study argues that inherently interpretable models can act as an empirical test bed for questions about cross-linguistic influence. It also suggests that ICALL systems and proficiency testing should take learners' societal language into account.

Key points

• The corpus has 472 Portuguese heritage-language texts averaging about 162 words, written by adolescents in Germany and the UK at CEFR levels B1, B2 and C1.
• The average importance of each feature class was: morphological 21.32%, lexical 20.34%, discursive 18.87%, syntactic 15.64%, count-based 13.47% and grammatical 10.37%.
• The strongest single predictors included:
  • irregular verbs in the imperfect indicative;
  • SUBTLEX contextual diversity and frequency measures;
  • Portuguese Vocabulary Profile (B1) words;
  • passive verbs;
  • first-person use.
• Learners in the English context used the passive more at every level, and the gap was widest at C1. Learners in the German context increasingly preferred the imperfect tense and first person as their proficiency rose.
• Lexical differences between the two groups shrank at higher proficiency, while morphosyntactic differences grew.

Interesting ideas

• I1. An EBM is additive, so every feature's contribution can be plotted as its own curve, which shows exactly why a decision was made and avoids an opaque ensemble.
• I2. The authors controlled for topic because n-gram classifiers can win by detecting subject matter rather than the property being studied, so features must validly represent the construct.
• I3. Clustering-based undersampling balances classes while still covering the different kinds of examples, instead of dropping items at random.
• I4. Feature effects were broken down by proficiency level, which revealed opposite trends: lexical differences converged while morphosyntactic ones diverged.
• I5. Rather than measuring each group against a monolingual norm, the study compared two bilingual groups directly, so variation was not framed as deficiency.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Additive shape-function models could predict when a parse or frame is likely wrong from interpretable features, so every repair rule comes with a readable reason.
• I2 (learn): This suggests a negative control in which test sets are topic-matched, so a gain counts only if it survives once content shortcuts are removed.
• I3 (learn): Silver data agreed on by the witness parsers could be clustered and then sampled, so rare constructions are not swamped by repetitive easy sentences.
• I4 (answer): Student-reader results on school tests could be split by grade or difficulty level, to see which skills plateau and which keep diverging.
• I5 (tree): When evaluating across UD dialects or treebanks, comparing variants against each other can expose systematic annotation preferences without declaring one of them the gold standard.