📄 Learning with learner corpora: Using the TLE for native language identification

Summary

The authors test whether the Treebank of Learner English (TLE) helps identify an author's native language from their English-language text (Native Language Identification, NLI). TLE has two parallel versions: the students' original sentences with errors and their corrected counterparts. MaltParser was trained on each version and on their mixture. For comparison, two more parsers were trained on the English Web Treebank (EWT): one on the full corpus and one on half of it. These parsers were used to parse essays from the TOEFL11 corpus. Dependency features were extracted from the parses, namely relations between lemmas and PoS tags, with and without labels, and fed to a linear SVM. The parser trained on the version with errors yielded slightly better classification than the one trained on the corrected version, even though its syntactic analysis is considerably worse, even on learner texts. Thus parsing accuracy and NLI accuracy are inversely related here: the parser's errors on ungrammatical constructions appear to carry a useful signal. The work shows that learner corpora are useful beyond syntactic analysis as well.

Key points

• TLE contains 5,124 sentences with errors, taken from the FCE corpus. It covers 10 native languages with 500 sentences each, split 4,124 / 500 / 500. Both the original and the corrected versions are annotated according to UD.
• The LAS of the model trained on the corrected TLE is 94.5 on the corrected test set and 90.1 on the original one. The model trained on the original TLE reaches only 85.7 and 85.1, and the model trained on EWT about 80.7.
• NLI was performed on TOEFL11 (12,100 essays, 11 native languages) using only syntactic features, of which there were on average more than 1.5 million. The classifier was LinearSVC with default parameters. PoS tags were assigned by HunPos trained on EWT.
• The model trained on the original TLE slightly outperforms the one trained on the corrected version. The full EWT performs on par with the best TLE model, while half of EWT performs noticeably worse. From this the authors conclude that both corpus size and domain matter.
• German is classified most accurately (81.5–82.5 %). The largest gap between models is for Arabic: 68.0 versus 66.1. In the authors' view, the models may have produced such similar results because they shared a PoS tagger trained on standard English.