Elamur Research⌕
☰
home / papers / 2012 / Automated Essay Scoring Based on Finite State Transducer: towards ASR Transcription of Oral English Speech

Automated Essay Scoring Based on Finite State Transducer: towards ASR Transcription of Oral English Speech

Paper · 2012 · Xingyuan Peng, Dengfeng Ke and Bo Xu

Summary

The paper addresses automatic content scoring of spoken ESL picture-description answers, where the text to be graded comes from an ASR system rather than from the learner's own writing. Standard AES techniques fit this setting poorly. Spelling, punctuation and grammar features become meaningless on noisy transcripts. Bag-of-words content models are also misled, because a strongly constrained language model tends to turn mumbled or off-topic speech into on-topic keywords. The authors instead model each exam topic with a Finite State Transducer. The FST maps word sequences to the picture they describe, so it captures both the order of the four pictures and the link between surface phrasing and meaning. Several FST-based features are combined by linear regression into a score. On 417 graded speeches, the FST approach clearly beats an LSA-SVR baseline, especially on ASR output (about 0.80 vs 0.58 correlation). Adding WordNet synonym similarity to substitution costs helps on manual transcripts. The result shows that sequence-aware matching holds up under recognition noise, while order-free content measures break down.

Key points

  • Data: 417 valid one-minute oral picture compositions from three ESL classes. Each was scored 0–20 (7 levels) by three raters, with an average inter-rater correlation of about 0.87.

  • Two transcript conditions:

    • Manual transcripts (CRR).
    • HTK-based ASR output, with an overall WER of 58.86% and a match rate of 72.88%. WER drops sharply for higher-scoring speakers, while the match rate stays fairly stable.

  • Features and scoring: 12 FST-derived features (ED, NED, MN, MR, CMV, SSF), each computed from two different FST building sets, are combined via linear regression trained with cross-validation.

  • Average correlations with expert scores:

    MethodCRRASR
    FST0.8980.804
    LSA-SVR0.8050.584
    Length feature0.7960.187

  • Improved FST: Synonym similarity from WordNet::Similarity, corrected by hand, raises the CRR score to 0.906. The ASR score barely changes (0.8045).

Interesting ideas

  • I1. Scoring content by aligning against an ordered reference structure, rather than counting keywords, penalises topic words strung together without sense. This matters whenever an upstream model may hallucinate plausible vocabulary.
  • I2. The FST layers surface wording under sense units (one per picture), separating how something is said from which content slot it fills.
  • I3. Synonym similarity enters as a softer substitution cost in the edit-distance-style match, not as a hard equivalence.
  • I4. The same system is evaluated on clean and noisy input side by side, with a trivial length feature as a control that exposes which methods collapse under noise.
  • I5. The authors note that strong language-model constraints make errors look topical, a reminder that noise from a confident upstream component is systematic rather than random.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Graders of student answers to tale or school questions could align the answer's event sequence with the expected one, instead of rewarding keyword overlap.
  • I2 (world): Event-order checks in the world of a text could match recovered events to expected story slots, tolerating varied phrasing while still enforcing temporal order.
  • I3 (frames): When matching frames, VerbNet-class or WordNet-close predicates could get graded substitution costs, keeping the match explainable as an edit path.
  • I4 (learn): Evaluations could routinely include a degraded-input condition and a shallow baseline as a negative control, to show that gains do not come from surface artefacts.
  • I5 (tree): Parser-output consumers should expect biased errors (e.g. attachments drifting toward frequent patterns) and treat agreement with priors as weaker evidence.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️Chinese Academy of SciencesBeijing · China194 works · 23,906 citations🔬Institute of AutomationBeijing · China70 works · 14,251 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,263 worksSpeech & spoken language · 480 worksGrammatical error correction & learner language · 156 works

Cite

Cite as Xingyuan Peng, Dengfeng Ke and Bo Xu. 2012. Automated Essay Scoring Based on Finite State Transducer: towards ASR Transcription of Oral English Speech. Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{peng2012automated,
  title = {{Automated Essay Scoring Based on Finite State Transducer: towards ASR Transcription of Oral English Speech}},
  author = {Xingyuan Peng, Dengfeng Ke and Bo Xu},
  booktitle = {Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2012},
  url = {https://aclanthology.org/P12-1006/}
}

References

25 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Cyril Allauzen, Michael Riley, Johan Schalkwyk, Wojciech Skut and Mehryar Mohri. 2007. OpenFst: a general and efficient weighted finite-state transducer library. In Proceedings of International Conference on Implementation and Application of Automata, 4783: 11-23. find
  2. Yigal Attali. 2011. A differential word use measure for content analysis in automated essay scoring. ETS research report, ETS RR-11-36. ↗
  3. Yigal Attali and Jill Burstein. 2006. Automated essay scoring with e-rater R V.2. The Journal of Technology, Learning, and Assessment, 4(3), 1-34. find
  4. Jill Burstein, Joel Tetreault, and Slava Andreyev. 2010. Using entity-based features to model coherence in student essays. In Human Language Technologies: The Annual Conference of the North American Chapter of the ACL, 681-684. find
  5. Chih-Chung Chang, Chih-Jen Lin. 2011. LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology, Vol. 2. find
  6. Yen-Yu Chen, Chien-Liang Liu, Chia-Hoang Lee, and Tao-Hsing Chang. 2010. An unsupervised automated essay scoring system. IEEE Intelligent Systems, 61-67. ↗
  7. Catia Cucchiarini, Helmer Strik, and Lou Boves. 2000. Quantitative assessment of second language learners’ fluency by means of automatic speech recognition technology. Acoustical Society of America, 107(2): 989-999. find
  8. Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer and Richard Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society for Information Science. 41(6): 391-407. find
  9. Shen Huang, Hongyan Li, Shijin Wang, Jiaen Liang and Bo Xu. 2010. Automatic reference independent evaluation of prosody quality using multiple knowledge fusions. In INTERSPEECH, 610-613. find
  10. Tsunenori Ishioka and Masayuki Kameda. 2004. Automated Japanese essay scoring system: jess. In Proceedings of the International Workshop on database and Expert Systems applications. ↗
  11. Tuomo Kakkonen, Niko Myller, Jari Timonen, and Erkki Sutinen. 2005. Automatic essay grading with probabilistic latent semantic analysis. In Proceedings of Workshop on Building Educational Applications Using NLP, 29-36. find
  12. Thomas K. Landauer, Darrell Laham and Peter Foltz. 2003. Automatic essay assessment. Assessment in Education: Principles, Policy and Practice (10:3), 295-309. ↗
  13. Deryle Lonsdale and Diane Strong-Krause. 2003. Automated rating of ESL essays. In Proceedings of the HLT-NAACL Workshop: Building Educational Applications Using NLP. find
  14. Andreas Maier, F. Hönig, V. Zeissler, Anton Batliner, E. Körner, N. Yamanaka, P. Ackermann, Elmar Nöth 2009. A language-independent feature set for the automatic evaluation of prosody. In INTERSPEECH, 600-603. find
  15. Mehryar Mohri. 2004. Weighted finite-state transducer algorithms: an overview. Formal Languages and Applications, 148 (620): 551-564. ↗
  16. Leonardo Neumeyer, Horacio Franco, Vassilios Digalakis, Mitchel Weintraub. 2000. Automatic scoring of pronunciation quality. Speech Communication, 30(2-3): 83-94. ↗
  17. Ted Pedersen, Siddharth Patwardhan and Jason Michelizzi. 2004. WordNet::Similarity - measuring the relatedness of concepts. In Proceedings of the National Conference on Artificial Intelligence, 144-152. find
  18. Xingyuan Peng, Dengfeng Ke, Zhenbiao Chen and Bo Xu. 2010. Automated Chinese essay scoring using vector space models. In Proceedings of IUCS, 149-153. find
  19. Isaac Persing, Alan Davis and Vincent Ng. 2010. Modeling organization in student essays. In Proceedings of EMNLP, 229-239.
  20. Lawrence M. Rudner and Tahung Liang. 2002. Automated essay scoring using Bayes’ theorem. The Journal of Technology, Learning and Assessment, 1(2):3-21. find
  21. G. Salton, C. Yang, A. Wong. 1975. A vector space model for automatic indexing. Communications of the ACM, 18(11): 613-620. ↗
  22. Alex J. Smola and Bernhard Schölkopf. 2004. A tutorial on support vector regression. Statistics and Computing 14(3): 199-222. ↗
  23. Robert A. Wagner and Michael J. Fischer. 1974. The string-to-string correction problem. Journal of the ACM, 21(1):168-173. ↗
  24. Helen Yannakoudakis, Ted Briscoe and Ben Medlock. 2011. A new dataset and method for automatically grading ESOL texts. In Proceedings of ACL, 180-189.
  25. Steve Young, Gunnar Evermann, Mark Gales, Thomas Hain, Dan Kershaw, Xunying Liu, Gareth Moore, Julian Odell, Dave Ollason, Dan Povey, Valtcho Valtchev, Phil Woodland. 2006. The HTK book (for HTK version 3.4). Cambridge University Engineering Department. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/peng-2012-automated-essay-scoring-based-finite/