Elamur Research⌕
☰
home / papers / 2014 / Lexical Chaining for Measuring Discourse Coherence Quality in Test-taker Essays

Lexical Chaining for Measuring Discourse Coherence Quality in Test-taker Essays

Summary

Automated essay assessment needs dependable signals of how coherent a text is, and earlier discourse features capture this only partly. Somasundaran, Burstein and Chodorow test whether lexical chains (sequences of semantically related nouns running through a text, after Morris and Hirst, 1991) can supply such signals. They reimplement the chaining algorithm of Hirst and St-Onge (1995), using Lin's thesaurus to grade how closely words are related, and derive two feature groups. LEX-1 describes chain counts, sizes, lexical variety and link types, which stand for topic unity, elaboration and vocabulary range. LEX-2 records whether chains begin, end or continue across explicit discourse cues found by an essay-oriented cue tagger, which stands for organization. Classifiers are trained on test-taker essays that were manually rated for coherence. The essays come from six writing tasks that differ in genre and writer population. On their own, the chain features outperform previously used discourse features. The strongest systems combine them with RST-parser features and grammar, usage and mechanics error features. The study shows that chain-based features help across several genres and writer groups.

Key points

  • Chains link nouns, adjective-noun and noun-noun units through three relation types, each defined by a Lin's thesaurus similarity threshold:
    • extra-strong: above 0.8 (repetition scores 1);
    • strong: above 0.172, searched within the previous six sentences;
    • medium-strong: above 0.099, searched within the previous three sentences.
  • LEX-1 contains 38 features, including length-normalized counts. A chain counts as "large" if it has more than four nodes, and it shows "variety" if it contains more than one word type.
  • LEX-2 has 138 features built on cue tags from the Burstein et al. (1998) tagger, such as arg-init:PARALLEL and arg-dev:CONTRAST. Each feature uses the full tag, its primary part or its secondary part, which reduces data sparseness.
  • The data covers six essay subsets, including persuasive/expository essays by graduate applicants (PE-G-N, 145 essays) and argument-critique essays (AC-G-N).
  • On one task, where the source reading and lecture already provide the content structure, the chain-feature system reached a QWK of 0.465. This beat gramErr+program but fell short of the best combination, in which RST features played a larger role.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

40 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 38 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{somasundaran2014lexical,
  title = {{Lexical Chaining for Measuring Discourse Coherence Quality in Test-taker Essays}},
  author = {Somasundaran, Swapna and Burstein, Jill and Chodorow, Martin},
  booktitle = {Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers},
  year = {2014},
  url = {https://aclanthology.org/C14-1090/}
}

References

44 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yigal Attali and Jill Burstein. 2006. Automated essay scoring with e-rater v. 2.0. Journal of Technology, Learning, and Assessment, 4:3. find
  2. Regina Barzilay and Michael Elhadad. 1997. Using lexical chains for text summarization. In Proceedings of the ACL workshop on intelligent scalable text summarization, volume 17, pages 10–17. find
  3. Regina Barzilay and Mirella Lapata. 2008. Modeling local coherence: An entity-based approach. Computational Linguistics, 34(1):1–34. find our card
  4. Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. 2012. An empirical investigation of statistical significance in NLP. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 995–1005. Association for Com… find
  5. Brent Bridgeman, Catherine Trapani, and Yigal Attali. 2012. Comparison of human and machine scoring of essays: Differences by gender, ethnicity, and country. Applied Measurement in Education, 25(1):27–40. find
  6. Jill Burstein, Karen Kukich, Susanne Wolff, Ji Lu, and Martin Chodorow. 1998. Enriching automated essay scoring using discourse marking. In Workshop on Discourse Relations and Discourse Marking. ERIC Clearinghouse. find
  7. Jill Burstein, Jane Shore, John Sabatini, Brad Moulder, Steven Holtzman, and Ted Pedersen. 2012. The language musesm system: Linguistically focused instructional authoring. Technical report, Educational Testing Services (ETS). find
  8. Jill Burstein, Joel Tetreault, and Martin Chodorow. 2013a. Holistic discourse coherence annotation for noisy essay writing. Dialogue and Discourse, 4(2):34–52. find our card
  9. Jill Burstein, Joel Tetreault, and Nitin Madnani, 2013b. Handbook of Automated Essay Evaluation: Current Applications and New Directions, chapter The E-rater Automated Essay Scoring System. Routledge. find
  10. Joseph Carthy and Michael Sherwood-Smith. 2002. Lexical chains for topic tracking. In 2002 IEEE International Conference on Systems, Man and Cybernetics, volume 7. IEEE. find
  11. Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4):213. find
  12. Paul Deane. 2013. On the relation between automated essay scoring and modern views of the writing construct. Assessing Writing, 18(1):7 – 24. Automated Assessment of Writing. find
  13. Micha Elsner and Eugene Charniak. 2008. Coreference-inspired coherence modeling. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Short Papers, pages 41–44. Association for Computational Linguistics. find our card
  14. Micha Elsner, Joseph Austerweil, and Eugene Charniak. 2007. A unified local and global model for discourse coherence. In Proceedings of NAACL/HLT. find our card
  15. Gonenc Ercan and Ilyas Cicekli. 2007. Using lexical chains for keyword extraction. Information Processing & Management, 43(6):1705–1714. find
  16. Lijun Feng, Noémie Elhadad, and Matt Huenerfauth. 2009. Cognitively motivated features for readability assessment. In Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics, pages 229–237. Association for Computational Linguistics. find
  17. Michael Flor, Beata Beigman Klebanov, and Kathleen M. Sheehan. 2013. Lexical tightness and text complexity. In Proceedings of the Workshop on Natural Language Processing for Improving Textual Accessibility, pages 29–38, Atlanta, Georgia, June. Association for Computational Linguistics. find
  18. Peter W Foltz, Walter Kintsch, and Thomas K Landauer. 1998. The measurement of textual coherence with latent semantic analysis. Discourse processes, 25(2-3):285–307. find
  19. Barbara J Grosz, Scott Weinstein, and Aravind K Joshi. 1995. Centering: A framework for modeling the local coherence of discourse. Computational Linguistics, 21(2):203–225. find our card
  20. Michael AK Halliday and Ruqaiya Hasan. 1976. Cohesion in english. English Language Series. Longman Group Ltd. find
  21. Paula Hatch, Nicola Stokes, and Joe Carthy. 2000. Topic detection, a new application for lexical chaining. In the proceedings of BCS-IRSG, pages 94–103. find
  22. Derrick Higgins, Jill Burstein, Daniel Marcu, and Claudia Gentile. 2004. Evaluating multiple aspects of coherence in student essays. In HLT-NAACL, pages 185–192. find
  23. Graeme Hirst and David St-Onge. 1995. Lexical chains as representations of context for the detection and correction of malapropisms. WordNet: An electronic lexical database, 305:305–332. find
  24. Jerry R Hobbs. 1979. Coherence and coreference. Cognitive Science, 3(1):67–90. find
  25. Ziheng Lin, Hwee Tou Ng, and Min-Yen Kan. 2011. Automatically evaluating text coherence using discourse relations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 997–1006. Association for Computational Linguisti… find our card
  26. Dekang Lin. 1998. Automatic retrieval and clustering of similar words. In Proceedings of the 17th international conference on Computational linguistics-Volume 2, pages 768–774. Association for Computational Linguistics. find
  27. William C Mann and Sandra A Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text, 8(3):243–281. find our card
  28. Daniel Marcu. 2000. The theory and practice of discourse parsing and summarization. MIT Press. find
  29. Eleni Miltsakaki and Karen Kukich. 2000. Automated evaluation of coherence in student essays. In Proceedings of LREC 2000. find
  30. Alden J Moe. 1979. Cohesion, coherence, and the comprehension of text. Journal of Reading, 23(1):16–20. find
  31. Dan Moldovan and Adrian Novischi. 2002. Lexical chains for question answering. In Proceedings of the 19th international conference on Computational linguistics-Volume 1, pages 1–7. Association for Computational Linguistics. find
  32. Jane Morris and Graeme Hirst. 1991. Lexical cohesion computed by thesaural relations as an indicator of the structure of text. Computational linguistics, 17(1):21–48. find our card
  33. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:282… find
  34. Charles A Perfetti and Alan M Lesgold. 1977. Discourse Comprehension and Sources of Individual Differences. ERIC. find
  35. Emily Pitler and Ani Nenkova. 2008. Revisiting readability: A unified framework for predicting text quality. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 186–195. Association for Computational Linguistics. find
  36. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind K Joshi, and Bonnie L Webber. 2008. The Penn Discourse TreeBank 2.0. In LREC. find our card
  37. Vasile Rus and Nobal Niraula. 2012. Automated detection of local coherence in short argumentative essays based on centering theory. In Computational Linguistics and Intelligent Text Processing, pages 450–461. Springer. find
  38. Mark D Shermis and Jill Burstein. 2013. Handbook of automated essay evaluation: Current applications and new directions. Routledge. find
  39. Radu Soricut and Daniel Marcu. 2006. Discourse generation using utility-trained coherence models. In Proceedings of the COLING/ACL on Main conference poster sessions, pages 803–810. Association for Computational Linguistics. find
  40. Nicola Stokes. 2003. Spoken and written news story segmentation using lexical chains. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology: Proceedings of the HLT-NAACL 2003 student research workshop-Volume… find
  41. Paul Van den Broek, Charles R Fletcher, and Kirsten Risden. 1993. Investigations of inferential processes in reading: A theoretical and methodological integration. Taylor & Francis. find
  42. Paul Van den Broek. 2012. Individual and developmental differences in reading comprehension: Assessing cognitive processes and outcomes. Measuring up: Advances in how we assess reading ability, page 39. find
  43. Y Wang, M Harrington, and P White. 2012. Detecting breakdowns in local coherence in the writing of Chinese English learners. Journal of Computer Assisted Learning, 28(4):396–410. find
  44. Stephen P Witte and Lester Faigley. 1981. Coherence, cohesion, and writing quality. College composition and communication, pages 189–204. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/somasundaran-2014-lexical-chaining-measuring-discourse-coherence/