Elamur Research⌕
☰
home / papers / 2026 / Towards Integrating Pose Estimation with Neuroimaging for the Analysis of Signed Language Video Stimuli

Towards Integrating Pose Estimation with Neuroimaging for the Analysis of Signed Language Video Stimuli

Paper · 2026 · Sébastien Vandenitte, Doris Hernández, Jarkko Keränen et al.

Summary

The paper describes a way to examine whether the movement characteristics of signed video stimuli shape how the brain processes them. Neuroimaging studies of sign languages often use video for ecological validity, but they usually treat motion variability as noise. The authors return to the 120 Finnish Sign Language (FinSL) clips from an earlier EEG detection study on enactment (constructed action) by Hernández et al. (2025). They run MediaPipe Pose Landmarker on each clip and keep only the 2D coordinates for the wrists, the nose and a shoulder-midpoint torso centroid. From these they derive smoothed kinematic measures. The approach does not time-align EEG and pose signals, as the MEG work of Rivolta et al. (2025) did. Instead, it builds a per-video kinematic profile and pairs it with ERP amplitudes averaged across participants for that same stimulus. The work is ongoing and exploratory, so its main contribution is the analysis pipeline and the combined data matrix. It also discusses the benefits and limits of reusing existing brain data for post-hoc analyses of stimulus kinematics.

Key points

  • Data: 120 FinSL video stimuli from Hernández et al. (2025). They are grouped into sets of four depicting one action, such as typing on a keyboard, at increasing degrees of enactment, from none to full-blown.
  • Pose estimation: MediaPipe Pose Landmarker was applied to all clips. The z coordinates were discarded for reliability, and Savitzky–Golay filters were used to reduce landmark noise.
  • Kinematic variables:
    • velocity and acceleration of the nose, upper torso, left wrist and right wrist;
    • distance between the two wrists;
    • area of the triangle formed by the nose and both wrists.
  • EEG integration: epochs were averaged across subjects for each stimulus, rather than per subject. This yields stimulus-level N2 and P3 mean amplitudes that are linked to the mean kinematic values.
  • Motivation: the variables follow Jantunen et al. (2020), which found head and torso velocity and acceleration differences in constructed action. They also follow the multi-articulator approach of Rivolta et al. (2025).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of JyväskyläJyväskylä · Finland4 works · 62,197 citations🎓University of NamurNamur · Belgium6 works · 21 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 worksPsychology · 1,552 worksSocial Psychology · 306 worksHuman-Computer Interaction · 141 works

Cite

Cite as Sébastien Vandenitte, Doris Hernández, Jarkko Keränen, Tommi Jantunen and Anna Puupponen. 2026. Towards Integrating Pose Estimation with Neuroimaging for the Analysis of Signed Language Video Stimuli. Proceedings of the LREC 2026 12th Workshop on the Representation and Processing of Sign Languages: Language in Motion.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2q3vp6hdx56b

@inproceedings{vandenitte2026integrating,
  title = {{Towards Integrating Pose Estimation with Neuroimaging for the Analysis of Signed Language Video Stimuli}},
  author = {Sébastien Vandenitte, Doris Hernández, Jarkko Keränen, Tommi Jantunen and Anna Puupponen},
  booktitle = {Proceedings of the LREC 2026 12th Workshop on the Representation and Processing of Sign Languages: Language in Motion},
  year = {2026},
  doi = {10.63317/2q3vp6hdx56b},
  url = {https://doi.org/10.63317/2q3vp6hdx56b}
}

References

19 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bauer, A., Kuder, A., Schulder, M. and Schepens, J. (2024). Phonetic differences between affirmative and feedback head nods in German Sign Language (DGS): A pose estimation study. Plos one, 19(5), e0304040. find
  2. Brookshire, G., Lu, J., Nusbaum, H. C., Goldin-Meadow, S. and Casasanto, D. (2017). Visual cortex entrains to sign language. Proceedings of the National Academy of Sciences, 114(24), 6352-6357. find
  3. Bux Sargano, A., Vandenitte, S., Jantunen, T. and Kimmelman, V. (2024). Evaluation of Head Pose Estimation Algorithms for Sign Language Analysis. In International Conference on IT and Industrial Technologies (ICIT), pages 1-6, Chiniot, Pakistan. https://doi.org/10.1109/ICIT63607.2024.10859 486 Bybee… — [doi:10.1109/icit63607.2024.10859](https://doi.org/10.1109/icit63607.2024.10859) ↗
  4. Diessel, H. (2017). Usage-based linguistics. In Mark Aronoff (Ed.), Oxford Research Encyclopedia of Linguistics. Oxford University Press, Oxford, United Kingdom. https://doi.org/10.1093/acrefore/97801993846 55.013.363 Drijvers, L., and Mazzini, S. (2023). Neural oscillations in audiovisual language… — [doi:10.1093/acrefore/97801993846](https://doi.org/10.1093/acrefore/97801993846) ↗
  5. HCII 2023. Lecture Notes in Computer Science, vol 14029, pages 402-438. Springer, Cham. https://doi.org/10.1007/978-3-031-35748-0_30 — [doi:10.1007/978-3-031-35748-0_30](https://doi.org/10.1007/978-3-031-35748-0_30) ↗
  6. Hernández, D., Puupponen, A., Keränen, J., Vandenitte, S., Anible, B., Ortega, G. and Jantunen, T. (2025). Neuroelectrical and behavioral correlates of constructed action recognition in Finnish sign language. Neuroscience, 575, 140-149. https://doi.org/10.1016/j.neuroscience.2025.03 .046 Hernández,… — [doi:10.1016/j.neuroscience.2025.03](https://doi.org/10.1016/j.neuroscience.2025.03) ↗
  7. Hernández, D., Puupponen, A., and Jantunen, T. (2022a). The contribution of event-related potentials to the understanding of sign language processing and production in the brain: Experimental evidence and future directions. Frontiers in Communication, 7, 750256. https://doi.org/10.3389/fcomm.2022.75… — [doi:10.3389/fcomm.2022.750256](https://doi.org/10.3389/fcomm.2022.750256) ↗
  8. Hernández, D., Puupponen, A., Keränen, J., Wainio, T., Pippuri, O., Ortega, G. and Jantunen, T. (2022b). Use of Sign Language Videos in EEG and MEG Studies : Experiences from a Multidisciplinary Project Combining Linguistics and Cognitive Neuroscience. In J. H. Jantunen, J. Kalja-Voima, M. Laukkarin… find
  9. Hosemann, J., Herrmann, A., Steinbach, M., Bornkessel-Schlesewsky, I. and Schlesewsky, M. (2013). Lexical prediction via forward models: N400 evidence from German Sign Language. Neuropsychologia, 51(11), 2224-2237. https://doi.org/10.1016/j.neuropsychologia.201 3.07.013 Hothorn, T., Hornik, K. and Z… — [doi:10.1016/j.neuropsychologia.201](https://doi.org/10.1016/j.neuropsychologia.201) ↗
  10. Jantunen, T., Weerdt, D., Burger, B. and Puupponen, A. (2020). The more you move, the more action you construct: A motion capture study on head and upper-torso movements in constructed action in Finnish Sign Language narratives. Gesture, 19(1), 72–96. https://doi.org/10.1075/gest.19042.jan — [doi:10.1075/gest.19042.jan](https://doi.org/10.1075/gest.19042.jan) ↗
  11. Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., Zhang, F., Chang, C., Yong, M.G., Lee, J., Chang, W., Hua, W., Georg, M. and Grundmann, M. (2019). MediaPipe: A Framework for Building Perception Pipelines. ArXiv, abs/1906.08172. — [arXiv:1906.08172](https://arxiv.org/abs/1906.08172) find
  12. Motamedi, Y., Schouwstra, M., Smith, K., Culbertson, J. and Kirby, S. (2019). Evolving artificial sign languages in the lab: From improvised gesture to systematic sign. Cognition, 192, 103964. find
  13. Pouw, W., Dingemanse, M., Motamedi, Y. and Özyürek, A. (2021). A Systematic Investigation of Gesture Kinematics in Evolving Manual Languages in the Lab. Cognitive Science, 45(7), e13014. https://doi.org/10.1111/cogs.13014 — [doi:10.1111/cogs.13014](https://doi.org/10.1111/cogs.13014) ↗
  14. Pouw, W. (2024). Wim Pouw's EnvisionBOX modules for social signal processing (Version 1.0.0) [Computer software]. https://github.com/WimPouw/envisionBOX_mo dulesWP Puce, A. and Hämäläinen, M. S. (2017). A review of issues related to data acquisition and analysis in EEG/MEG studies. Brain sciences, 7… ↗
  15. Rivolta, C. L., Lizarazu, M., Costello, B., and Carreiras, M. (2025). Cortical tracking of sign language: The role of language knowledge in tracking of different articulators. Proceedings of the National Academy of Sciences, 122(51), e2512665122. find
  16. Stamp, R., Cohn, D., Hel-Or, H., and Sandler, W. (2024). Kinect-ing the Dots: Using Motion-Capture Technology to Distinguish Sign Language Linguistic From Gestural Expressions. Language and Speech, 67(1), 255–276. https://doi.org/10.1177/00238309231169502 — [doi:10.1177/00238309231169502](https://doi.org/10.1177/00238309231169502) ↗
  17. Trettenbrein, P. C., and Zaccarella, E. (2021). Controlling video stimuli in sign language and gesture research: The OpenPoseR package for analyzing OpenPose motion-tracking data in R. Frontiers in Psychology, 12, 628728. find
  18. West, B. T., Welch, K. B., and Galecki, A. T. (2022). Linear mixed models: a practical guide using statistical software. Chapman and Hall/CRC. https://doi.org/10.1201/9781003181064 — [doi:10.1201/9781003181064](https://doi.org/10.1201/9781003181064) ↗
  19. Wilkinson, M., Dumontier, M., Aalbersberg, I. et al. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Sci Data 3, 160018 https://doi.org/10.1038/sdata.2016.18 — [doi:10.1038/sdata.2016.18](https://doi.org/10.1038/sdata.2016.18) ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vandenitte-2026-integrating-pose-estimation-neuroimaging-analysis/