Elamur Research⌕
☰
home / papers / 2013 / A User Study: Technology to Increase Teachers' Linguistic Awareness to Improve Instructional Language Support for English Language Learners

A User Study: Technology to Increase Teachers' Linguistic Awareness to Improve Instructional Language Support for English Language Learners

Paper · 2013 · Jill Burstein, John Sabatini, Jane Shore et al.

Summary

Content-area teachers in U.S. K-12 schools increasingly teach English language learners (ELLs) but often lack training in spotting the linguistic features that make classroom texts hard for these students. The authors present Language Muse, a web-based teacher professional development (TPD) application from ETS. Alongside training videos and readings, it has a module that uses natural language processing to automatically flag potentially unfamiliar language at four levels: vocabulary, phrases, sentences and discourse. Teachers then use this feedback to build lesson plans with language-based scaffolding, activities and assessments. The tool rests on two ideas. First, teachers must recognise sources of difficulty before they can support learners. Second, finding such features by hand takes too long, even for linguistically aware teachers. In pilots, teachers took assessments before and after using the tool. According to the authors, their linguistic knowledge and awareness grew, and they became better at designing suitable language instruction. That instruction drew on the system's feedback. The work matters because it applies NLP to support the human reader's teacher, rather than to produce a single readability score.

Key points

  • The ELL population in U.S. public schools grew by 51% between 1997 and 2009. Many of these students sit in mainstream subject classes without extra English instruction.
  • Language Muse is a TPD application. Its automated feedback covers vocabulary, phrasal, sentential and discourse levels.
  • The design is grounded in research on several ELL difficulties:
    • Tier 1/2/3 academic vocabulary
    • morphology and cognates (e.g., English/Spanish)
    • polysemous content words
    • figurative terms and cultural references
    • long multi-clause sentences
    • mixed discourse relations, anaphora and connectors
  • Evaluation used pre-post teacher assessments in a pilot TPD setting. The paper reports gains in teachers' linguistic knowledge and in the quality of their ELL-oriented instruction.
  • The related work contrasts the tool with two other systems:
    • REAP, which uses language-model readability to select texts for ESL students
    • WERTi, which highlights problematic forms such as determiners, prepositions, wh-questions and phrasal verbs for learners

Interesting ideas

  • I1. Instead of collapsing difficulty into one grade-level number, the system marks specific features at separate linguistic levels, so a human can see why a text is hard.
  • I2. Difficulty is treated as relative to the reader: what is easy for a native speaker on grade level may be opaque to an ELL reading below grade level.
  • I3. Common words with a specialised domain sense, like "prime" in maths, invite a misleading everyday reading.
  • I4. Readers must track reference across pronouns, synonyms, paraphrases and demonstratives like "this" or "those", and connectors such as "because" or "therefore" build cohesion.
  • I5. Success is judged by measuring human users before and after they use the tool, not only by system accuracy.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Mova's readers could produce a level-by-level difficulty report on a school text (word, phrase, clause, discourse) that points to tree and frame evidence, making their reading of the text explainable to a teacher.
  • I2 (learn): evaluation sets could be split by reader profile (e.g., figurative compounds, cultural references, polysemous domain terms) to show which phenomena Mova itself fails on, rather than reporting one average score.
  • I3 (world): the compiled ontology could flag domain-shifted senses from WordNet when a text's topic is maths or science, so word problems do not bind the everyday meaning.
  • I4 (frames): coreference could treat demonstratives that point back to whole ideas as a distinct case, and connectors could be mapped to explicit cause or contrast links between frames.
  • I5 (learn): explanation outputs could be tested by checking whether a person answers text questions better after seeing Mova's tree-and-frame explanation, giving an outcome-based check on explainability.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🤝Educational Testing ServicePrinceton · United States27 works · 2,212 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Arts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksLiterature and Literary Theory · 519 worksEducation · 203 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksGrammatical error correction & learner language · 156 works

Cite

Cite as Jill Burstein, John Sabatini, Jane Shore, Brad Moulder and Jennifer Lentini. 2013. A User Study: Technology to Increase Teachers' Linguistic Awareness to Improve Instructional Language Support for English Language Learners. Proceedings of the Workshop on Natural Language Processing for Improving Textual Accessibility.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{burstein2013user,
  title = {{A User Study: Technology to Increase Teachers' Linguistic Awareness to Improve Instructional Language Support for English Language Learners}},
  author = {Jill Burstein, John Sabatini, Jane Shore, Brad Moulder and Jennifer Lentini},
  booktitle = {Proceedings of the Workshop on Natural Language Processing for Improving Textual Accessibility},
  year = {2013},
  url = {https://aclanthology.org/W13-1501/}
}

References

52 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adger, C. T., Snow, C., & Christian D. (2002). What teachers need to know about language. Washington, DC: Center for Applied Linguistics. find
  2. August, D. (2003). Supporting the development of English literacy in English language learners: Key issues and promising practices (Report No. 61). Baltimore, MD: Johns Hopkins University Center for Research on the Education of Students Placed at Risk. find
  3. Barzilay, Regina and Mirella Lapata (2008). ‘Modeling Local Coherence: An Entity-Based Approach.’ Computational Linguistics, 43(1): 1-34. ↗
  4. Beck, I. L., McKeown, M. G., & Kucan, L. (2008). Creating robust vocabulary: Frequently asked questions and extended examples. New York, NY: Guilford Press. find
  5. Benson, M., Benson, E., & Ilson, R. (Eds.). (1997). The BBI Combinatory Dictionary of English: A Guide to Word Combinations. Amsterdam & Philadelphia: John Benjamins Publishing Company. find
  6. Berninger, V., Abbot, R., Nagy, W., & Carlisle, J. (2009). Growth in phonological, orthographic, and morphological awareness in grades 1-6. Journal of Psycholinguistic Research, 39, 141-163. find
  7. Breland, H. Jones, R., and Jenkins, L (1994). The college board vocabulary study. ↗
  8. Burstein, J., Shore, J., Sabatini, J., Moulder, B., Holtzman, S., & Pedersen, T. (2012). The Language Muse system: Linguistically focused instructional authoring ETS RR-12-21. Princeton, NJ: ETS. find
  9. Burstein, J., and Pedersen, T. (2010). Towards Improving Synonym Options in a Text Modification Application. University of Minnesota Supercomputing Institute Research Report Series, UMSI 2010/165, November 2010. find
  10. Burstein, J., Chodorow, M., and Leacock, C. (2004). Automated Essay Evaluation: The Criterion Online Service, AI Magazine, 25(3), 27-36. find
  11. Burstein, J., Kukich, K., Wolff, S., Lu, C., Chodorow, M., Braden-Harder, L., and Harris, M. D. (1998). Automated Scoring Using A Hybrid Feature Identification Technique. In the Proceedings of the Annual Meeting of the Association of Computational Linguistics, August, 1998. Montreal, Canada. find
  12. Calderón, M. (2007). Teaching reading to English language learners, grades 6-12: A framework for improving achievement in the content areas. Thousand Oaks, CA: Corwin Press. find
  13. Calderón, M., August, D., Slavin, R., Cheung, A., Durán, D., & Madden, N. (2005). Bringing words to life in classrooms with English language learners. In A. Hiebert & M. Kamil (Eds.), Research and development on vocabulary. Mahwah, NJ: Lawrence Erlbaum Associates. find
  14. Carlo, M. S., August, D., McLaughlin, B., Snow, C. E., Dressler, C., Lippman, D. N., & White, C. E. (2004). Closing the gap: Addressing the vocabulary needs of English language learners in bilingual and mainstream classrooms. Reading Research Quarterly, 39, 188-215. ↗
  15. Coleman, D., & Pimentel, S. (2011a). Publishers’ criteria for the Common Core State Standards in English Language Arts and Literacy, grades 3-12. Washington, DC: National Governors Association Center for Best Practices and Council of Chief State School Officers. find
  16. Collins-Thompson, Kevyn and Jamie Callan (2004). ‘A Language Modeling Approach to Predicting Reading Difficulty.’ In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics. Boston, MA: Association for Computational Linguistics, 193-200.
  17. Darling-Hammond, L. (2000). Teacher quality and student achievement: A review of state policy evidence. Education Policy Analysis Archives, 8. find
  18. Flesch, R.. (1948). A new readability yardstick. Journal of Applied Psychology, 32, 221-233. ↗
  19. Flinspach, S. L., Scott, J. A., Samway, K. D., & Miller, T. (2008, March). Developing cognate awareness to enhance literacy: Importante y necesario. Paper presented at the Annual Meeting of the American Educational Research Association, New York, NY.. find
  20. Francis, D., August, D. Goldenberg, C., & Shanahan, T. (2004). Developing literacy skills in English language learners: Key issues and promising practices. Retrieved June 11, 2007, from: www.cal.org/natl-lit-panel/reports/Executive_Summary.pdf Futagi, Y., Deane, P., Chodorow, M., & Tetreault, J. (2008). A Computational Approach to Detecting Collocation Errors in the Writing of Non-native Speakers of English, Computer Assisted Language Learning, Vol. 21, pp. 353–367. find
  21. Gándara, P., Maxwell-Jolly, J., & Driscoll, A. (2005). Listening to teachers of English language learners: A survey of California teachers’ challenges, experiences, and professional development needs. Sacramento, CA: The Regents of the University of California. Retrieved from http://www.cftl.org/documents/2005/listeningforweb. pdf. ↗
  22. Goldenberg, C. (2008). Teaching English language learners: What the research does—and does not— say. American Educator, 32, 8-21. find
  23. Goldman, S. R., & Rakestraw Jr., J. A. (2000). Structural aspects of constructing meaning from text. In M. L. Kamil, P. B. Mosenthal, P. D. Pearson, & R. Barr (Eds.), Handbook of reading research (Vol. III, pp. 311-335). Mahwah, NJ: Lawrence Erlbaum Associates. ↗
  24. Graesser, Arthur C., Danielle S. McNamara, and Jonna M. Kulikowich (2011). ‘Coh-Metrix: Providing Multilevel Analyses of Text Characteristics.’ Educational Researcher, 40(5): 223-234. find
  25. Green, C., Foote, M., Walker, C., & Shuman, C. (2010). From questions to answers: Education faculty members learn about English learners. In S. Szabo, M. B. Sampson, M. M. Foote, & F. Falk-Ross (Eds.), Mentoring literacy professionals: Continuing the spirit of CRA/ALER after 50 years (pp. 113-125). Commerce, TX: Texas A&M University Press. find
  26. Heilman, Michael, Lee Zhao, Juan Pinto, and Maxine Eskenazi (2008). ‘Retrieval of Reading Materials for Vocabulary and Reading Practice.’ In Proceedings of the Third Workshop on Innovative Use of NLP for Building Educational Applications. Columbus, OH: Association for Computational Linguistics, 80-88. find
  27. Kieffer, M. J. & Lesaux, N. K. (2008). The role of derivational morphology in the reading comprehension of Spanish-speaking English language learners. Reading and Writing, 21, 783-804. ↗
  28. Kintsch, W. (1998). Comprehension: A paradigm for comprehension. Cambridge, UK: Cambridge University Press. find
  29. Leacock, C. & Chodorow, M. (2003). C-rater: Scoring of Short-Answer Questions. Computers and the Humanities, Vol. 37, pp. 389–405. find
  30. Lesaux, N. K., Kieffer, M. J., Faller, S. E., & Kelley, J. G. (2010). The effectiveness and ease of implementation of an academic vocabulary intervention for linguistically diverse students in urban middle schools. Reading Research Quarterly, 45, 196-228. ↗
  31. Lin, Dekang (1998). ‘Automatic Retrieval and Clustering of Similar Words.’ In “Proceedings of the 17 th International Conference on Computational Linguistics and the 36 th Annual Meeting of the Association for Computational Linguistics. Montreal, Canada: 768-774.
  32. Marcu, Daniel (1999). ‘Discourse Trees Are Good Indicators of Importance in Text. In Advances in Automatic Text Summarization, eds. Inderjeet Mani and Mark T. Maybury. Cambridge, MA: MIT Press, 123-136. find
  33. Meurers, W. Detmar, Ramon Ziai, Luiz Amaral, Adriane Boyd, Aleksandar Dimitrov, Vanessa Metcalf, and Niels Ott (2010). ‘Enhancing Authentic Web Pages for Language Learners.’ In Proceedings of the NAACL HLT 2010 Fifth International Workshop on Innovative Use of NLP for Building Educational Applications, eds. Joel Tetreault, Jill Burstein, and Claudia Leacock. Los Angeles, CA: Association for Computational Linguistics, 10-18. find
  34. Meyer, B. J. F. (2003). Text coherence and readability. Topics in Language Disorders, 23, 204-221. ↗
  35. Mihalcea, Rada, Ravi Sinha, and Diana McCarthy (2010). ‘SemEval-2010 Task 2: Cross-Lingual Lexical Substitution.’ In Proceedings of SemEval-2010: Fifth International Workshop on Semantic Evaluations. Uppsala, Sweden: Association for Computational Linguistics, 9-14. find
  36. Miller, George A. (1990). ‘An On-line Lexical Database.’ International Journal of Lexicography 3(4): 235-312. find
  37. Miltsakaki, Eleni (2009). ‘Matching Readers’ Preferences and Reading Skills with Appropriate Web Texts.’ In Proceedings of the European Association for Computational Linguistics. Athens, Greece: Association for Computational Linguistics, 49-52. find
  38. Nagy, W., Beringer, V., & Abbott, R. (2006). Contributions of morphology beyond phonology to literacy outcomes of upper elementary and middle school students. Journal of Educational Psychology, 98, 134-147. ↗
  39. National Clearinghouse for English Language Acquisition (2011). The growing numbers of English learner students. Washington, DC: Author. Retrieved from http://www.ncela.gwu.edu/files/uploads/9/growingLE P_0809.pdf. ↗
  40. National Governors Association Center for Best Practices and Council of Chief State School Officers (2010). Common Core State Standards for English language Arts & Literacy in History/Social Studies, Science, and Technical Subjects. Appendix A: Research supporting key elements of the Standards. Washington, DC: Author. find
  41. Nelson, Jessica, Charles Perfetti, David Liben, and Meredith Liben (2012). Measures of Text Difficulty: Testing Their Predictive Value for Grade Levels and Student Performance. Washington, DC: The Council of Chief State School Officers. Retrieved from http://www.ccsso.org/Documents/2012/Measures%2 0ofText%20Difficulty_final.2012.pdf. ↗
  42. Pappamihiel, N. E., Lake, V., & Rice, D. (2005). Adapting a Social Studies lesson to include English language learners. Social Studies and the Young Learner, 17, 4-7. find
  43. Peske, H. G., & Haycock, K. (2006). Teaching inequality: How poor and minority students are shortchanged on teacher quality. Washington, DC: The Education Trust. Retrieved from http://www.edtrust.org/sites/edtrust.org/files/publicati ons/files/TQReportJune2006.pdf. ↗
  44. Pitler, Emily and Ani Nenkova (2008). ‘Revisiting Readability: A Unified Framework for Predicting Text Quality.’ In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing. Honolulu, HI: Association for Computational Linguistics, 186-195.
  45. Proctor, C. P., Dalton, D., Uccelli, P., Biancarosa, G., Mo, E., Snow, C. E., & Neugebauer, S. (2011). Improving comprehension online (ICON): Effects of deep vocabulary instruction with bilingual and monolingual fifth graders. Reading and Writing: An Interdisciplinary Journal, 24, 517-544. find
  46. Proctor, C. P., Dalton, B., & Grisham, D. (2007). Scaffolding English language learners and struggling readers in a multimedia hypertext environment with embedded strategy instruction and vocabulary support. Journal of Literacy Research, 39, 71-93. find
  47. Rivera, M. O., Moughamian, A. C., Lesaux, N. K., & Francis, D. J. (2008). Language and reading interventions for English language learners and English language learners with disabilities. Portsmouth, NJ: Research Corporation, Center on Instruction. find
  48. Rutherford William E. and Michael Sharwood Smith (1985). ‘Consciousness-Raising and Universal Grammar.’ Applied Linguistics 6(3): 274-282. ↗
  49. Schwarm, Sarah E. and Mari Ostendorf (2005). ‘Reading Level Assessment Using Support Vector Machines and Statistical Language Models.’ In Proceedings of the Annual Meeting of the Association for Computational Linguistics. Ann Arbor, MI: Association for Computational Linguistics, 523-530.
  50. Schleppegrell, M. J. (2007). The linguistic challenges of mathematics teaching and learning: A research review. Reading and Writing Quarterly, 23, 139-159. ↗
  51. Schleppegrell, M. J., & de Oliveira, L. C. (2006). An integrated language and content approach for history teachers. Journal of English for Academic Purposes, 5, 254-268. ↗
  52. Walqui, A., & Heritage, M. (2012, January). Instruction for diverse groups of ELLs. Paper presented at the Understanding Language Conference, Stanford, CA. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/burstein-2013-user-study-technology-increase-teachers/