Elamur Research⌕
☰
home / papers / 2013 / Recognizing English Learners' Native Language from Their Writings

Recognizing English Learners' Native Language from Their Writings

Paper · 2013 · Baoli Li

Summary

This paper describes a system for the first Native Language Identification (NLI) shared task, NLI2013. The task is to guess an English learner's first language from an essay, which can help second language teaching and forensic authorship profiling. The data is TOEFL11, a corpus of TOEFL essays covering 11 L1s, released by ETS through the LDC. The author treats the problem as plain supervised text classification. Each essay becomes a TFIDF-weighted vector, and a linear SVM (Libsvm) is trained on it. The features are deliberately shallow: lowercased tokens, word 2–3-grams, character 2–5-grams, simple style counts and the essay metadata. Syntactic features were left out on purpose, so the study measures how far these traditional cues go on the first large open dataset. The full-feature run reached 0.773 accuracy, placing 18th of 29 teams in the closed track. Ablations show that the feature types help different L1s unevenly, which points toward class-specific feature sets. Character n-grams emerge as cheap but strong features, while style and metadata features add almost nothing.

Key points

  • TOEFL11 is described as 12,100 essays of roughly 300–400 words, split 9,900 train / 1,100 dev / 1,100 test. The eleven native languages span five Asian ones (Chinese, Japanese, Korean, Hindi and Telugu), four European ones (French, German, Italian and Spanish), plus Arabic and Turkish, and every essay is also tagged with a proficiency level (low/medium/high) and a Prompt ID.
  • Four official runs were submitted:
    • HAUTCS-1, all features: 0.773.
    • HAUTCS-2, without character n-grams: 0.758.
    • HAUTCS-3, which also drops style: 0.760.
    • HAUTCS-4, words plus metadata only: 0.756.
  • HAUTCS-1 was accidentally trained without the dev set. Retraining it on train+dev gave 0.781 after the evaluation.
  • Per-language F1 for the best run peaked on German (about 0.90) and was lowest on Hindi (about 0.69). 10-fold cross-validation on train+dev gave 0.799, higher than the test score.
  • A words-only run (HAUTCS-5) matched HAUTCS-4 at 0.756, so the metadata gave no measurable gain. Adding style features slightly lowered accuracy.

Interesting ideas

  • I1. Character n-grams are taken over the whole sentence, with spaces replaced by a marker symbol and explicit start/end symbols added, so the n-grams capture affixes and cross-word spellings without a morphological analyzer.
  • I2. Feature groups are added run by run and per-class F1 is checked, revealing that each group helps only some classes (character n-grams suit Spanish, plain words suit Telugu).
  • I3. A words-only run is compared against words+metadata to test whether the metadata carries any signal, and it turns out to carry none.
  • I4. Paragraph-based style features were exposed to a train/test format mismatch, because the original test release had paragraph breaks stripped.
  • I5. Cross-validation accuracy (0.799) ran noticeably above held-out test accuracy (0.773), a reminder that in-distribution folds can be optimistic.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): A boundary-aware character n-gram model could serve as a backoff signal for the tagger and lemmatizer on unknown, misspelled or learner-influenced word forms.
  • I2 (learn): Evaluation reports should break gains down per category, e.g. per dependency relation or per frame type, rather than relying on a single headline accuracy.
  • I3 (learn): This is a cheap negative control to adopt: retrain with one feature source removed and confirm the gain is real before keeping that source.
  • I4 (tokens): Keep paragraph and sentence boundary conventions identical across all corpora fed to the tokenizer, and flag inputs where structural markers are missing.
  • I5 (learn): Silver-data training should always be scored on a truly separate test set as well as internal folds, and the gap should be tracked.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksSocial Psychology · 306 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Baoli Li. 2013. Recognizing English Learners' Native Language from Their Writings. Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{li2013recognizing,
  title = {{Recognizing English Learners' Native Language from Their Writings}},
  author = {Baoli Li},
  booktitle = {Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications},
  year = {2013},
  url = {https://aclanthology.org/W13-1715/}
}

References

19 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahn, C. S. 2011. Automatically Detecting Authors' Native Language. Master's thesis, Naval Postgraduate School, Monterey, CA. find
  2. Bergsma, S., Post, M., and Yarowsky, D. 2012. Stylometric analysis of scientific articles. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 327–337, Montréal, Canada. Association for Computational Linguistics. find
  3. Blanchard, D., Tetreault, J., Higgins, D., Cahill, A., and Chodorow, M. 2013. TOEFL11: A Corpus of Non-Native English. Technical report, Educational Testing Service. ↗
  4. Brooke, J. and Hirst, G. 2012a. Measuring interlanguage: Native language identification with l1-influence metrics. In Calzolari, N., Choukri, K., Declerck, T., Dogan, M. U., Maegaard, B., Mariani, J., Odijk, J., and Piperidis, S., editors, Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012), pages 779–784, Istanbul, Turkey Brooke, J. and Hirst, G. 2012b. Robust, Lexicalized Native Language Identification. In Proceedings of COLING 2012, pages 391-408, Mumbai, India. find
  5. Bykh, S. and Meurers, D. 2012. Native Language Identification using Recurring n-grams - Investigating Abstraction and Domain Dependence. In Proceedings of COLING 2012, pages 425-440, Mumbai, India. find
  6. Chang, C.-C. and Lin C.-J. 2011. LIBSVM : a library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:3:27:1-27. find
  7. Crossley, S. A. and McNamara, D. 2012. Detecting the First Language of Second Language Writers Using Automated Indices of Cohesion, Lexical Sophistication, Syntactic Complexity and Conceptual Knowledge. In Jarvis, S. and Crossley, S. A., editors, Approaching Language Transfer through Text Classification, pages 106-126. Multilingual Matters. ↗
  8. Jarvis, S., Castañeda-Jiménez, G., and Nielsen, R. 2012. Detecting L2 Writers' L1s on the Basis of Their Lexical Styles. In Jarvis, S. and Crossley, S. A., editors, Approaching Language Transfer through Text Classification, pages 34-70. Multilingual Matters. ↗
  9. Jarvis, S. and Paquot, M. 2012. Exploring the Role of n-Grams in L1 Identification. In Jarvis, S. and Crossley, S. A., editors, Approaching Language Transfer through Text Classification, pages 71-105. Multilingual Matters. ↗
  10. Kochmar, E. 2011. Identification of a writer’s native language by error analysis. Master’s thesis, University of Cambridge. find
  11. Koppel, M., Schler, J., and Zigdon, K. 2005. Determining an author’s native language by mining a text for errors. In Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, pages 624–628, Chicago, IL. ACM. find
  12. Koppel, M., Schler, J., and Argamon, S. 2008. Computational methods in authorship attribution. Journal of the American Society for information Science and Technology, 60(1):9–26. ↗
  13. Li, B., and Vogel, C. 2010. Improving Multiclass Text Classification with Error-Correcting Output Coding and Sub-class Partitions. In Proceedings of the 23rd Canadian Conference on Artificial Intelligence, pages 4-15, Ottawa, Canada. find
  14. Swanson, B. and Charniak, E. 2012. Native language detection with tree substitution grammars. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 193–197, Jeju Island, Korea. find
  15. Tetreault, J., Blanchard, D., and Cahill, A. 2013. A report on the first native language identification shared task. In Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications. Atlanta, GA, USA. find
  16. Tofighi, P.; Köse, C.; and Rouka, L. 2012. Author’s native language identification from web-based texts. International Journal of Computer and Communication Engineering. 1(1):47-50 find
  17. Torney, R.; Vamplew, P.; and Yearwood, J. 2012. Using psycholinguistic features for profiling first language of authors. Journal of the American Society for Information Science and Technology. 63(6):1256-1269. ↗
  18. Wong, S.-M. J. and Dras, M. 2011. Exploiting Parse Structures for Native Language Identification. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1600–1610, Edinburgh, Scotland, UK. find
  19. Wong, S.-M. J., Dras, M., and Johnson, M. 2012. Exploring Adaptor Grammars for Native Language Identification. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 699–709, Jeju Island, Korea. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2013-recognizing-english-learners-native-language/