Elamur Research⌕
☰
home / papers / 2026 / Predicting Gaze Location without Camera or Eye-Tracker

Predicting Gaze Location without Camera or Eye-Tracker

Paper · 2026 · Saman Rezapoor, Sajad Shirali-Shahreza and Gerald Penn

Summary

Eye-trackers and webcams can estimate where a reader is looking on a web page, but most users lack such devices or decline to be filmed for privacy reasons. The authors test whether multimodal LLMs can infer gaze position from screen content and cursor activity alone. They sample screenshots from The Eye Of The Typer (EOTT) dataset and draw mouse paths and clicks onto each image. Each frame then goes to Gemini 3 Flash or GPT-5.2 Instant. The system prompt encodes known web-reading habits, such as F-pattern and Z-pattern scanning, plus a temporal smoothing rule. Two setups are compared: one where the model predicts gaze through a whole video without any true gaze, and one where the true gaze from the preceding frame is supplied as an anchor. Accuracy is modest. Supplying the anchor clearly helps, raising region accuracy from about 16–18% to about 27%. Blank or calibration pages cause large errors. The authors argue that small on-device models could eventually make this a hardware-free and less intrusive way to estimate gaze.

Key points

  • Data: 3 randomly selected participants (P1, P2, P7) from EOTT's 51, with 50 screenshots each taken at 5-second intervals. The reference gaze for each frame is the geometric median, computed with Weiszfeld's algorithm, of up to 5 gaze samples (recorded at 60 Hz) within ±500 ms of the frame.
  • Method: mouse traces are drawn onto screenshots with OpenCV. The model returns structured JSON containing coordinates, an attention-pattern label, a reasoning-mode label and a confidence score.
  • Model comparison: for Gemini 3 Flash vs. GPT-5.2 Instant, region accuracy was 15.7% vs. 17.7% (full-video) and 27.5% vs. 27.4% (per-frame). Mean pixel error was about 357/356 px (full-video) and 290/286 px (per-frame).
  • Per-user averages: mean error fell from 380 to 330 px, and region accuracy rose from 17.0% to 24.2%, when moving from full-video to per-frame. P7 was notably worse than the other two participants. Restricting to high-confidence predictions did not improve results.
  • Coarse accuracy: on a grid of 12 regions of 300×300 px, the predicted region was the correct one or a neighbouring one in over half of the cases. Error spikes coincided with page navigation and empty or calibration screens.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TorontoToronto · Canada96 works · 142,007 citations🎓Amirkabir University of TechnologyTehran · Iran2 works · 4 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 worksInformation Systems · 1,437 worksHuman-Computer Interaction · 141 works

Cite

Cite as Saman Rezapoor, Sajad Shirali-Shahreza and Gerald Penn. 2026. Predicting Gaze Location without Camera or Eye-Tracker. Proceedings fo the Second International Workshop on Eye-Tracking Resources and Evaluation for Human-Aligned NLP.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2hpa22f63k2t

@inproceedings{rezapoor2026predicting,
  title = {{Predicting Gaze Location without Camera or Eye-Tracker}},
  author = {Saman Rezapoor, Sajad Shirali-Shahreza and Gerald Penn},
  booktitle = {Proceedings fo the Second International Workshop on Eye-Tracking Resources and Evaluation for Human-Aligned NLP},
  year = {2026},
  doi = {10.63317/2hpa22f63k2t},
  url = {https://doi.org/10.63317/2hpa22f63k2t}
}

References

8 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexandra Papoutsaki, Aaron Gokaslan, James Tompkin, Yuze He, and Jeff Huang. 2018. The eye of the typer: a benchmark and analysis of gaze behavior during typing. In Proceedings of ACM ETRA. find
  2. Seyed Fatemeh Ahmadzadeh. 2024. User’s gaze estimation based on the movement of the mouse and screen information. Bachelor’s thesis, Amirkabir University of Technology, Tehran, IRAN, June. find
  3. Alexandra Papoutsaki, Patsorn Sangkloy, James Laskey, Nediyana Daskalova, Jeff Huang, and James Hays. 2016. Webgazer: scalable webcam eye tracking using user interactions. In Proceedings of IJCAI, pages 3839–3845. AAAI Press. find
  4. Diego Alves. 2025. Benchmarking language model surprisal for eye-tracking predictions in Brazilian Portuguese. In Proceedings of the First International Workshop on Gaze Data and Natural Language Processing, pages 7–17, Varna, Bulgaria. INCOMA Ltd., Shoumen, BULGARIA. find
  5. Cristina Maria Popescu and Sergiu Nisioi. 2025. Exploring mouse tracking for reading on Romanian data. In Proceedings of the First International Workshop on Gaze Data and Natural Language Processing, pages 44–51, Varna, Bulgaria. IN-COMA Ltd., Shoumen, BULGARIA. find
  6. Amir Beck and Shoham Sabach. 2015. Weiszfeld’s method: Old and new results. J. Optimization Theory and Applications, 164(1):1–40. find
  7. Georg Buscher, Edward Cutrell, and Meredith Ringel Morris. 2009. What do you see when you’re surfing? using eye tracking to predict salient regions of web pages. In Proceedings of CHI, pages 21–30. find
  8. Mads Soegaard. 2021. Visual hierarchy: Organizing content to follow natural eye movement patterns. IxDF - Interaction Design Foundation (accessed 29th March, 2026). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rezapoor-2026-predicting-gaze-location-without-camera/