Elamur Research⌕
☰
home / papers / 2023 / GazeReader: Detecting Unknown Word Using Webcam for English as a Second Language (ESL) Learners

GazeReader: Detecting Unknown Word Using Webcam for English as a Second Language (ESL) Learners

Paper · 2023 · Jiexin Ding, Bowen Zhao, Yuqi Huang et al.

Summary

GazeReader asks whether an ordinary laptop webcam, instead of a costly dedicated eye tracker, can reveal which words an English learner does not know while reading. Webcam gaze is too imprecise for classic cues such as fixation counts or regressions. The authors therefore let the text itself do much of the work. They recorded 12 graduate-student ESL readers on 36 TOEFL passages with WebGazer.js. Readers marked unfamiliar words after reading each passage. The noisy gaze trace was smoothed, resampled and roughly aligned to word positions. Two LSTMs encode the gaze and word coordinates, and an attention-like product links the two. RoBERTa supplies contextual embeddings, and the model is enriched with term frequency, part-of-speech and named-entity features. The model reaches 98.09% accuracy and 75.73% F1, and does slightly better on unseen users. Ablations, however, show that the text signal carries almost all of the performance and the gaze adds little. A follow-up questionnaire found that learners want help mainly with proper nouns, words with several senses, and long, intricate sentences.

Key points

  • Data: 12 participants, 36 TOEFL articles (about 386 words each, 13,902 words in total), about four hours of reading per person. The study yielded 4,274 unknown-word clicks covering 1,232 distinct words.
  • Pre-processing: gaze was smoothed with a moving average (window 50) and resampled to 20 Hz with linear interpolation. Words were aligned to time by assuming a constant reading pace, and the context was the text seen within about 1 s. Negatives were created by pairing gaze with passages the user was not reading.
  • Main results: precision 71.21 and recall 80.70 give an F1 of 75.73. On held-out users the F1 is 78.26 ± 4.53. On held-out documents it falls to 56.31 ± 3.38.
  • Ablation (F1):
    • Removing context awareness (RoBERTa) collapses F1 to 10.00.
    • Removing gaze gives 75.59, almost no change.
    • Removing the knowledge features gives 74.93.
  • User study: reading assistance scored 4.35/5 and vocabulary management 3.93/5, both significantly above neutral. Privacy and power use were not rated as significant concerns.

Interesting ideas

  • I1. Removing each input stream in turn exposed that the advertised signal (gaze) barely matters while the text encoder does nearly everything, which is an honest way to check what a system actually relies on.
  • I2. Splitting evaluation into user-independent and document-independent settings showed that generalizing to new texts is far harder than generalizing to new people (F1 56 vs 78).
  • I3. Cheap lexical signals (term frequency, POS, NER) were placed next to a contextual encoder and gave a small but consistent gain for spotting hard words.
  • I4. Negative examples were made by pairing a real signal with text the reader was not looking at, giving mismatch cases for free.
  • I5. Learners reported that proper nouns, polysemous words and long, complex sentences cause the most trouble.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Run the same modality-dropping test on witness parsers and LLM teacher tiers. This acts as a negative control showing which source really drives agreement on silver data.
  • I2 (learn): Evaluations should hold out whole documents or tales, not just sentences. This keeps scores from hiding memorization of particular texts.
  • I3 (tokens): The tagger and lemmatizer could output a transparent rarity/difficulty flag from frequency, POS and entity type. Later stages could use it to explain which words need glossing.
  • I4 (learn): Self-play could generate negatives by attaching frames or answers to the wrong sentence. This would train and test whether components truly check the text they are given.
  • I5 (back): The text → graph → text path could target exactly these cases. It could resolve entities, add sense hints, and split long sentences into simpler clauses when it regenerates text for learners.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States353 works · 200,043 citations🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 worksHuman-Computer Interaction · 141 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksGrammatical error correction & learner language · 156 works

Cite

Cite as Jiexin Ding, Bowen Zhao, Yuqi Huang, Yuntao Wang and Yuanchun Shi. 2023. GazeReader: Detecting Unknown Word Using Webcam for English as a Second Language (ESL) Learners. arXiv:2303.10443.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2303.10443 · arXiv 2303.10443

@misc{ding2023gazereader,
  title = {{GazeReader: Detecting Unknown Word Using Webcam for English as a Second Language (ESL) Learners}},
  author = {Jiexin Ding, Bowen Zhao, Yuqi Huang, Yuntao Wang and Yuanchun Shi},
  year = {2023},
  doi = {10.48550/arxiv.2303.10443},
  eprint = {2303.10443},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.10443}
}

References

29 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Maria Barrett, Joachim Bingel, Nora Hollenstein, Marek Rei, and Anders Søgaard. 2018. Sequence Classification with Human Attention. In Proceedings of the 22nd Conference on Computational Natural Language Learning. Association for Computational Linguistics, Brussels, Belgium, 302–312. https://doi.org/10.18653/v1/K18-1030 ↗
  2. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. " O’Reilly Media, Inc.". find
  3. Kiyomi Chujo and Kathryn Oghigian. 2009. How many words do you need to know to understand TOEIC, TOEFL & EIKEN? An examination of text coverage and high frequency vocabulary. Journal of Asia TEFL 6, 2 (2009). find
  4. Charles Clifton Jr, Adrian Staub, and Keith Rayner. 2007. Eye movements in reading words and sentences. Eye movements (2007), 341–371. ↗
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 4171–4186.
  6. Yo Ehara, Nobuyuki Shimizu, Takashi Ninomiya, and Hiroshi Nakagawa. 2010. Personalized Reading Support for Second-Language Web Documents by Collective Intelligence. In Proceedings of the 15th International Conference on Intelligent User Interfaces (Hong Kong, China) (IUI ’10). Association for Computing Machinery, New York, NY, USA, 51–60. https://doi.org/10.1145/1719970.1719978 ↗
  7. Utpal Garain, Onkar Pandit, Olivier Augereau, Ayano Okoso, and Koichi Kise. 2017. Identification of Reader Specific Difficult Words by Analyzing Eye Gaze and Document Content. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), Vol. 01. 1346–1351. https://doi.org/10.1109/ICDAR.2017.221 ↗
  8. Riku Higashimura, Andrew Vargo, Motoi Iwata, and Koichi Kise. 2022. Helping Mobile Learners Know Unknown Words through Their Reading Behavior. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI EA ’22). Association for Computing Machinery, New York, NY, USA, Article 249, 5 pages. https://doi.org/10.1145/3491101.3519620 ↗
  9. Rui Hiraoka, Hiroki Tanaka, Sakriani Sakti, Graham Neubig, and Satoshi Nakamura. 2016. Personalized Unknown Word Detection in Non-Native Language Reading Using Eye Gaze. In Proceedings of the 18th ACM International Conference on Multimodal Interaction (Tokyo, Japan) (ICMI ’16). Association for Computing Machinery, New York, NY, USA, 66–70. https://doi.org/10.1145/2993148.2993167 ↗
  10. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9, 8 (1997), 1735–1780. ↗
  11. Nora Hollenstein and Ce Zhang. 2019. Entity Recognition at First Sight: Improving NER with Eye Movement Information. In North American Chapter of the Association for Computational Linguistics. ↗
  12. Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020. spaCy: Industrial-strength natural language processing in python. (2020). find
  13. Stephen Hutt and Sidney K. D’Mello. 2022. Evaluating Calibration-Free Webcam-Based Eye Tracking for Gaze-Based User Modeling. In Proceedings of the 2022 International Conference on Multimodal Interaction (Bengaluru, India) (ICMI ’22). Association for Computing Machinery, New York, NY, USA, 224–235. https://doi.org/10.1145/3536221.3556580 ↗
  14. Aulikki Hyrskykari. 2006. Eyes in attentive interfaces: Experiences from creating iDict, a gaze-aware reading aid. Tampere University Press. find
  15. Marcel A Just and Patricia A Carpenter. 1980. A theory of reading: from eye fixations to comprehension. Psychological review 87, 4 (1980), 329. ↗
  16. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019). ↗
  17. Yunfei Long, Rong Xiang, Qin Lu, Chu-Ren Huang, and Minglei Li. 2021. Improving Attention Model Based on Cognition Grounded Data for Sentiment Analysis. IEEE Transactions on Affective Computing 12, 4 (2021), 900–912. https://doi.org/10.1109/TAFFC.2019.2903056 ↗
  18. Tobias Lunte and Susanne Boll. 2020. Towards a Gaze-Contingent Reading Assistance for Children with Difficulties in Reading. In Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility (Virtual Event, Greece) (ASSETS ’20). Association for Computing Machinery, New York, NY, USA, Article 83, 4 pages. https://doi.org/10.1145/3373625.3418014 ↗
  19. Ahmad Azman Mokhtar and Rafizah Mohd Rawian. 2012. Guessing word meaning from context has its limit: Why. International Journal of Linguistics 4, 2 (2012), 288–305. ↗
  20. Paul Nation. 1990. Teaching and learning vocabulary. In Handbook of Practical Second Language Teaching and Learning. Routledge, 397–408. find
  21. Alexandra Papoutsaki, James Laskey, and Jeff Huang. 2017. SearchGazer: Webcam Eye Tracking for Remote Studies of Web Search. In Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval (Oslo, Norway) (CHIIR ’17). Association for Computing Machinery, New York, NY, USA, 17–26. https://doi.org/10.1145/3020165.3020170 ↗
  22. Alexandra Papoutsaki, Patsorn Sangkloy, James Laskey, Nediyana Daskalova, Jeff Huang, and James Hays. 2016. WebGazer: Scalable Webcam Eye Tracking Using User Interactions. In Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI). AAAI, 3839–3845. find
  23. Keith Rayner. 1998. Eye movements in reading and information processing: 20 years of research. Psychological bulletin 124, 3 (1998), 372. ↗
  24. John Read. 1988. Measuring the vocabulary knowledge of second langauge learners. RELC journal 19, 2 (1988), 12–25. ↗
  25. John Read. 2000. Assessing vocabulary. Cambridge university press. find
  26. Pat Rigg. 1991. Whole language in TESOL. Tesol Quarterly 25, 3 (1991), 521–542. ↗
  27. Norbert Schmitt, Diane Schmitt, and Caroline Clapham. 2001. Developing and exploring the behaviour of two new versions of the Vocabulary Levels Test. Language testing 18, 1 (2001), 55–88. ↗
  28. Yuntao Wang, Jiexin Ding, Ishan Chatterjee, Farshid Salemi Parizi, Yuzhou Zhuang, Yukang Yan, Shwetak Patel, and Yuanchun Shi. 2022. FaceOri: Tracking Head Position and Orientation Using Ultrasonic Ranging on Earphones. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 290, 12 pages. https://doi.org/10.1145/3491102.3517698 ↗
  29. Yuzhou Zhuang, Yuntao Wang, Yukang Yan, Xuhai Xu, and Yuanchun Shi. 2021. ReflecTrack: Enabling 3D Acoustic Position Tracking Using Commodity Dual-Microphone Smartphones. In The 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’21). Association for Computing Machinery, New York, NY, USA, 1050–1062. https://doi.org/10.1145/3472749.3474805 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ding-2023-gazereader-detecting-unknown-word-using/