Elamur Research⌕
☰
home / papers / 2025 / Unknown Word Detection for English as a Second Language (ESL) Learners Using Gaze and Pre-trained Language Models

Unknown Word Detection for English as a Second Language (ESL) Learners Using Gaze and Pre-trained Language Models

Paper · 2025 · Jiexin Ding, Bowen Zhao, Yuntao Wang et al.

Summary

Readers of English as a second language often get stuck on words they don't know. A system that notices this while they read could show a definition or explanation at that moment. The authors present EyeLingo, which treats unknown-word detection as a two-modality task combining eye gaze and text.

The gaze trajectory is used only to pick out the region of interest: a box of nearby words the reader is probably looking at. Inside that window, RoBERTa supplies contextual embeddings, which are combined with word-level knowledge features such as term frequency and part-of-speech. A transformer encoder-decoder with cross-attention links gaze positions to the text layout. Everything is trained jointly on data collected from 20 participants.

EyeLingo reaches 97.6% accuracy and 71.1% F1 with a professional eye tracker. It stays strong with cheap webcam gaze and runs in under a second. Ablations show that the language model does most of the work, while gaze adds per-user, moment-specific signal. This matters because detection becomes practical on ordinary hardware and with normal text formatting.

Key points

  • Offline results with a professional eye tracker: 97.6% accuracy, 71.1% F1.
  • With noisy webcam gaze: 97.3% accuracy, 65.1% F1.
  • Live prototype: about 56.5% F1, latency within 1 second. Users found it more useful and were more willing to use it than click-to-translate.
  • Earlier baselines:
    • iDict caught 36.5% of unknown words.
    • An SVM using gaze plus word length and rarity (Hiroka et al.) reached F1 55.6%.
    • Garain et al. reported F1 86%, but only for a single user.
    • These earlier setups used line spacing of 3.0–6.0, while everyday reading uses 1.0–2.0.
  • Eye-tracker accuracy is a hard limit:
    • Normal text needs roughly 0.3–0.6° accuracy, while typical trackers achieve 0.2–1.1°.
    • Participants reported errors spanning 1–3 lines even after calibration.
    • This is why baselines that rely only on gaze performed poorly.

Interesting ideas

  • I1. Use the imprecise signal only to narrow the search to a window of candidates, then let a strong text model make the per-word decision; this tolerates localization noise instead of fighting it.
  • I2. Feed explicit word-level knowledge (term frequency, part-of-speech) into the model next to contextual transformer embeddings, rather than hoping the PLM encodes it implicitly.
  • I3. Run ablations to measure how much each input modality contributes, and treat low reliance on a noisy input as a robustness feature rather than a failure.
  • I4. When exact alignment is impossible, label at the span (bounding-box) level and openly accept the resulting label noise, instead of forcing point-to-token mappings.
  • I5. Detecting which words a particular reader likely doesn't know enables personalized vocabulary lists and simplified notes.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): coreference could use a cheap, rough cue such as distance or salience to pick a candidate window, then let explainable syntactic and semantic checks choose the antecedent inside it.
  • I2 (tree): gparse could take lexicon frequency and tagger POS as extra input embeddings, which also makes it easier to trace why the parser chose a given attachment.
  • I3 (learn): evaluation could routinely zero out or scramble one input, such as one witness parser or one feature group, as a negative control that shows what each component really adds.
  • I4 (learn): when building silver data from witnesses whose tokenizations disagree, align at the span level and record possible mismatch as an explicit noise flag.
  • I5 (back): the generator could use a difficulty estimate from frequency and context to swap rare words for simpler ones when producing text for learners.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States353 works · 200,043 citations🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🎓University of Chinese Academy of SciencesBeijing · China147 works · 8,480 citations🎓Rice UniversityHouston · United States11 works · 1,850 citations

Also on the byline: Qinghai University.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 worksHuman-Computer Interaction · 141 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksGrammatical error correction & learner language · 156 works

Cite

Cite as Jiexin Ding, Bowen Zhao, Yuntao Wang, Xinyun Liu, Rui Hao, Ishan Chatterjee and Yuanchun Shi. 2025. Unknown Word Detection for English as a Second Language (ESL) Learners Using Gaze and Pre-trained Language Models. arXiv:2502.10378.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2502.10378 · arXiv 2502.10378

@misc{ding2025unknown,
  title = {{Unknown Word Detection for English as a Second Language (ESL) Learners Using Gaze and Pre-trained Language Models}},
  author = {Jiexin Ding, Bowen Zhao, Yuntao Wang, Xinyun Liu, Rui Hao, Ishan Chatterjee and Yuanchun Shi},
  year = {2025},
  doi = {10.48550/arxiv.2502.10378},
  eprint = {2502.10378},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2502.10378}
}

References

46 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Maria Barrett, Joachim Bingel, Nora Hollenstein, Marek Rei, and Anders Søgaard. 2018. Sequence Classification with Human Attention. In Proceedings of the 22nd Conference on Computational Natural Language Learning. Association for Computational Linguistics, Brussels, Belgium, 302–312. https://doi.org/10.18653/ v1/K18-1030 ↗
  2. Maria Barrett and Nora Hollenstein. 2020. Sequence labelling and sequence classification with gaze: Novel uses of eye-tracking data for Natural Language Processing. Language and Linguistics Compass 14, 11 (2020), 1–16. ↗
  3. R. Bates and H. O. Istance. 2003. Why Are Eye Mice Unpopular? A Detailed Comparison of Head and Eye Controlled Assistive Technology Pointing Devices. Univers. Access Inf. Soc. 2, 3 (oct 2003), 280–290. https://doi.org/10.1007/s10209-003-0053-y ↗
  4. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. " O’Reilly Media, Inc.". find
  5. Carmen Bisogni, Michele Nappi, Genoveffa Tortora, and Alberto Del Bimbo. 2024. Gaze analysis: A survey on its applications. Image and Vision Computing 144 (2024), 104961. https://doi.org/10.1016/j.imavis.2024.104961 ↗
  6. Robert Bixler and Sidney D’Mello. [n. d.]. Automatic gaze-based user-independent detection of mind wandering during computerized reading. 26, 1 ([n. d.]), 33–68. https://doi.org/10.1007/s11257-015-9167-1 ↗
  7. Zhaokang Chen and Bertram E. Shi. 2019. Using Variable Dwell Time to Accelerate Gaze-Based Web Browsing with Two-Step Selection. International Journal of Human–Computer Interaction 35, 3 (2019), 240–255. https://doi.org/10.1080/ 10447318.2018.1452351 arXiv:https://doi.org/10.1080/10447318.2018.1452351 ↗
  8. Shiwei Cheng, Zhiqiang Sun, Lingyun Sun, Kirsten Yee, and Anind K. Dey. [n. d.]. Gaze-Based Annotations for Reading Comprehension. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul Republic of Korea, 2015-04-18). ACM, 1569–1572. https://doi.org/10.1145/2702123.2702271 ↗
  9. Kiyomi Chujo and Kathryn Oghigian. 2009. How many words do you need to know to understand TOEIC, TOEFL & EIKEN? An examination of text coverage and high frequency vocabulary. Journal of Asia TEFL 6, 2 (2009). find
  10. Shuwen Deng, David R. Reich, Paul Prasse, Patrick Haller, Tobias Scheffer, and Lena A. Jäger. 2023. Eyettention: An Attention-based Dual-Sequence Model for Predicting Human Scanpaths during Reading. Proc. ACM Hum.-Comput. Interact. 7, ETRA, Article 162 (May 2023), 24 pages. https://doi.org/10.1145/3591131 ↗
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 4171–4186.
  12. Piaoyang Du, Wei Guo, and Shiwei Cheng. [n. d.]. Using eye-tracking for real-time translation: a new approach to improving reading experience. 6, 2 ([n. d.]), 150–164. https://doi.org/10.1007/s42486-024-00150-3 ↗
  13. Neeru Dubey, Simran Setia, Amit Arjun Verma, and S. R.S. Iyengar. 2020. WikiGaze: Gaze-Based Personalized Summarization of Wikipedia Reading Session. In Proceedings of the 3rd Workshop on Human Factors in Hypertext (Virtual Event, USA) (HUMAN’20). Association for Computing Machinery, New York, NY, USA, Article 4, 9 pages. https://doi.org/10.1145/3406853.3432662 ↗
  14. Paola Dussias. 2010. Uses of Eye-Tracking Data in Second Language Sentence Processing Research. Annual Review of Applied Linguistics 30 (03 2010), 149 – 166. https://doi.org/10.1017/S026719051000005X ↗
  15. Yo Ehara, Nobuyuki Shimizu, Takashi Ninomiya, and Hiroshi Nakagawa. 2010. Personalized Reading Support for Second-Language Web Documents by Collective Intelligence. In Proceedings of the 15th International Conference on Intelligent User Interfaces (Hong Kong, China) (IUI ’10). Association for Computing Machinery, New York, NY, USA, 51–60. https://doi.org/10.1145/1719970.1719978 ↗
  16. Utpal Garain, Onkar Pandit, Olivier Augereau, Ayano Okoso, and Koichi Kise. 2017. Identification of Reader Specific Difficult Words by Analyzing Eye Gaze and Document Content. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), Vol. 01. 1346–1351. https://doi.org/10.1109/ ICDAR.2017.221 ↗
  17. Wei Guo and Shiwei Cheng. 2019. An Approach to Reading Assistance with Eye Tracking Data and Text Features. In Adjunct of the 2019 International Conference on Multimodal Interaction (Suzhou, China) (ICMI ’19). Association for Computing Machinery, New York, NY, USA, Article 7, 7 pages. https://doi.org/10.1145/ 3351529.3360659 ↗
  18. Brett J. Hashimoto and Jesse Egbert. 2019. More Than Frequency? Exploring Predictors of Word Difficulty for Second Language Learners. Language Learning 69, 4 (2019), 839–872. https://doi.org/10.1111/lang.12353 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/lang.12353 ↗
  19. Riku Higashimura, Andrew Vargo, Motoi Iwata, and Koichi Kise. 2022. Helping Mobile Learners Know Unknown Words through Their Reading Behavior. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI EA ’22). Association for Computing Machinery, New York, NY, USA, Article 249, 5 pages. https://doi.org/10.1145/3491101.3519620 ↗
  20. Rui Hiraoka, Hiroki Tanaka, Sakriani Sakti, Graham Neubig, and Satoshi Nakamura. 2016. Personalized Unknown Word Detection in Non-Native Language Reading Using Eye Gaze. In Proceedings of the 18th ACM International Conference on Multimodal Interaction (Tokyo, Japan) (ICMI ’16). Association for Computing Machinery, New York, NY, USA, 66–70. https://doi.org/10.1145/2993148.2993167 ↗
  21. Nora Hollenstein, Marius Troendle, Ce Zhang, and Nicolas Langer. 2020. ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotation. In Proceedings of the Twelfth Language Resources and Evaluation Conference, Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (Eds.). European Language Resources Association, Marseille, France, 138–146. https://aclanthology.org/2020.lrec-1.18 ↗
  22. Nora Hollenstein and Ce Zhang. 2019. Entity Recognition at First Sight: Improving NER with Eye Movement Information. In North American Chapter of the Association for Computational Linguistics. ↗
  23. Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020. spaCy: Industrial-strength natural language processing in python. (2020). find
  24. Aulikki Hyrskykari. 2006. Eyes in attentive interfaces : experiences from creating iDict, a gaze-aware reading aid /. (01 2006). find
  25. Marcel A Just and Patricia A Carpenter. 1980. A theory of reading: from eye fixations to comprehension. Psychological review 87, 4 (1980), 329. ↗
  26. Jiajia Li, Grace Ngai, Hong Va Leong, and Stephen Chan. [n. d.]. Multimodal human attention detection for reading. In Proceedings of the 31st Annual ACM Symposium on Applied Computing (Pisa Italy, 2016-04-04). ACM, 187–192. https://doi.org/10.1145/2851613.2851681 ↗
  27. Change Liu, Chun Yu, Xiangyang Wang, Jianxiao Jiang, Tiaoao Yang, Bingda Tang, Yingtian Shi, Chen Liang, and Yuanchun Shi. 2024. CalibRead: Unobtrusive Eye Tracking Calibration from Natural Reading Behavior. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 8, 4, Article 154 (Nov. 2024), 30 pages. https://doi.org/10.1145/3699737 ↗
  28. Jiahui Liu, Jiannan Chi, Huijie Yang, and Xucheng Yin. 2022. In the eye of the beholder: A survey of gaze tracking techniques. Pattern Recognition 132 (2022), 108944. https://doi.org/10.1016/j.patcog.2022.108944 ↗
  29. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019). ↗
  30. Yunfei Long, Rong Xiang, Qin Lu, Chu-Ren Huang, and Minglei Li. 2021. Improving Attention Model Based on Cognition Grounded Data for Sentiment Analysis. IEEE Transactions on Affective Computing 12, 4 (2021), 900–912. https://doi.org/10.1109/TAFFC.2019.2903056 ↗
  31. Ahmad Azman Mokhtar and Rafizah Mohd Rawian. 2012. Guessing word meaning from context has its limit: Why. International Journal of Linguistics 4, 2 (2012), 288–305. ↗
  32. Paul Nation. 1990. Teaching and learning vocabulary. In Handbook of Practical Second Language Teaching and Learning. Routledge, 397–408. find
  33. Ayano Okoso, Takumi Toyama, Kai Kunze, Joachim Folz, Marcus Liwicki, and Koichi Kise. [n. d.]. Towards Extraction of Subjective Reading Incomprehension: Analysis of Eye Gaze Features. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (Seoul Republic of Korea, 2015-04-18). ACM, 1325–1330. https://doi.org/10.1145/2702613.2732896 ↗
  34. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research 21, 1 (2020), 5485–5551. ↗
  35. Keith Rayner. 1995. Eye Movements and Cognitive Processes in Reading, Visual Search, and Scene Perception. In Eye Movement Research, John M. Findlay, Robin Walker, and Robert W. Kentridge (Eds.). Studies in Visual Information Processing, Vol. 6. North-Holland, 3–22. https://doi.org/10.1016/S0926-907X(05)80003-0 ↗
  36. Keith Rayner. 1998. Eye movements in reading and information processing: 20 years of research. Psychological bulletin 124, 3 (1998), 372. ↗
  37. Pat Rigg. 1991. Whole language in TESOL. Tesol Quarterly 25, 3 (1991), 521–542. ↗
  38. Charles Sanches, Olivier Augereau, and Koichi Kise. 2016. Vertical error correction of eye trackers in nonrestrictive reading condition. IPSJ Transactions on Computer Vision and Applications 8 (12 2016). https://doi.org/10.1186/s41074-016-0008-x ↗
  39. Charles Lima Sanches, Olivier Augereau, and Koichi Kise. [n. d.]. Using the Eye Gaze to Predict Document Reading Subjective Understanding. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR) (Kyoto, 2017-11). IEEE, 28–31. https://doi.org/10.1109/ICDAR.2017.377 ↗
  40. Norbert Schmitt, Diane Schmitt, and Caroline Clapham. 2001. Developing and exploring the behaviour of two new versions of the Vocabulary Levels Test. Language testing 18, 1 (2001), 55–88. ↗
  41. John L. Sibert, Mehmet Gokturk, and Robert A. Lavine. [n. d.]. The reading assistant: eye gaze triggered auditory prompting for reading remediation. In Proceedings of the 13th annual ACM symposium on User interface software and technology - UIST ’00 (San Diego, California, United States, 2000). ACM Press, 101–107. https://doi.org/10.1145/354401.354418 ↗
  42. Ekta Sood, Simon Tannert, Philipp Mueller, and Andreas Bulling. 2020. Improving Natural Language Processing Tasks with Human Gaze-Guided Neural Attention. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 6327–6341. https://proceedings.neurips.cc/paper_files/paper/2020/ file/460191c72f67e90150a093b4585e7eb4-Paper.pdf ↗
  43. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (Long Beach, California, USA) (NIPS’17). Curran Associates Inc., Red Hook, NY, USA, 6000–6010.
  44. Yuntao Wang, Jiexin Ding, Ishan Chatterjee, Farshid Salemi Parizi, Yuzhou Zhuang, Yukang Yan, Shwetak Patel, and Yuanchun Shi. 2022. FaceOri: Tracking Head Position and Orientation Using Ultrasonic Ranging on Earphones. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 290, 12 pages. https://doi.org/10.1145/3491102.3517698 ↗
  45. Akito Yamaya, Goran Topić, Pascual Martínez-Gómez, and Akiko Aizawa. 2015. Dynamic-Programming–Based Method for Fixation-to-Word Mapping. In Intelligent Decision Technologies, Rui Neves-Silva, Lakhmi C. Jain, and Robert J. Howlett (Eds.). Springer International Publishing, Cham, 649–659. find
  46. Francesca Zermiani, Prajit Dhar, Ekta Sood, Fabian Kögel, Andreas Bulling, and Maria Wirzberger. 2024. InteRead: An Eye Tracking Dataset of Interrupted Reading. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (Eds.). ELRA and ICCL, Torino, Italia, 9154–9169. https://aclanthology.org/2024.lrec-main.802 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ding-2025-unknown-word-detection-english-second/