A Sentence Judgment System for Grammatical Error Detection
Summary
Learners of Chinese as a foreign language (CFL) have far fewer automatic grammar-checking tools than learners of English, even as MOOCs increase the need for large-scale assessment. The authors present a web-based demo system that decides whether a Chinese sentence contains grammatical errors. Each input is segmented, with unknown-word merging to handle OOV items, and POS-tagged. Two detectors then analyse it. The first applies 142 expert-written syntactic rules derived from Cheng's (1997) collection of 900 typical learner errors. The second is a statistical component. Bi- and tri-gram SRILM models trained on the Sinica corpus score 19,080 erroneous HSK sentences and their manual corrections, and normal densities fitted to these scores act as "correct" and "incorrect" models. A sentence is flagged when the incorrect model gives it higher probability. In tests, the rules proved precise but missed most errors, while the n-gram models showed the opposite pattern. Combining the two lets users trade recall against precision. Because the interface also displays the matched rules and n-gram frequencies, it can support autonomous learning.
Key points
- The rule notation uses wildcards, exclusions, alternatives, mutual exclusivity (mx{W1 W2}) and a follows-after operator over POS tags.
- The test set has 880 erroneous sentences from CSL learners at the NCKU Chinese Language Center, plus their 880 corrected versions.
- Rule-based detection reached a precision of 0.857 but a recall of only 0.224, with a false positive rate of 0.038.
- The 3-gram model outperformed the 2-gram model on every metric (F1 0.689 vs 0.638), but it wrongly flagged about 60% of correct sentences.
- Combining rules with AND raised precision to 0.924 and lowered FPR to 0.007, though recall fell to 0.083. The OR combination reached a recall of 1.0. The demo itself flags a sentence only when both methods agree.
Cite
@inproceedings{lee2014sentence,
title = {{A Sentence Judgment System for Grammatical Error Detection}},
author = {Lee, Lung-Hao and Yu, Liang-Chih and Lee, Kuei-Ching and Tseng, Yuen-Hsien and Chang, Li-Ping and Chen, Hsin-Hsi},
booktitle = {Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: System Demonstrations},
year = {2014},
url = {https://aclanthology.org/C14-2015/}
}References
- Andreas Stolcke. 2002. SRILM — An extensible language modeling toolkit. Proceedings of ICSLP’02, pages 901-904. find
- Chi-Hsin Yu and Hsin-Hsi Chen. 2012. Detecting word ordering errors in Chinese sentences for learning Chinese as a foreign language. Proceedings of COLING’12, pages 3003-3018. find
- Christopher D. Manning and Hinrich Schütze. 1999. Foundations of Statistical Natural Language Processing. MIT Press. Cambridge, MA. find
- Chung-Hsien Wu, Chao-Hung Liu, Matthew Harris and Liang-Chih Yu. 2010. Sentence correction incorporating relative position and parse template language model. IEEE Transactions on Audio, Speech, and Language Processing, 18(6):1170-1181. find
- Keh-Jiann Chen and Wei-Yun Ma. 2002. Unknown word extraction for Chinese documents. Proceedings of COLING’02, pages 169-175. find
- M. Cheng. 1997. Analysis of 900 Common Erroneous Samples of Chinese Sentences - for Chinese Learners from English Speaking Countries (in Chinese). Beijing, CN: Sinolingua. find
- Ru-Ying Chang, Chung-Hsien Wu, and Philips K. Prasetyo. 2012. Error diagnosis of Chinese sentences using inductive learning algorithm and decomposition-based testing mechanism. ACM Transactions on Asian Language Information Processing, 11(1):Article 3. find
- Yu-Fang Tsai and Keh-Jiann Chen. 2004. Reliable and cost-effective pos-tagging. International Journal of Computational Linguistics and Chinese Language Processing, 9(1):83-96. find
- Recall 0.224 0.751 0.838 1.000 1.000 0.083 0.083 F1 0.356 0.638 0.689 0.667 0.668 0.153 0.153 False Positive Rate 0.038 0.603 0.595 1.000 0.993 0.007 0.007 find