Elamur Research⌕
☰
home / papers / 2006 / Gesture Improves Coreference Resolution

Gesture Improves Coreference Resolution

Summary

Most coreference research uses written text. Spontaneous face-to-face explanations are disfluent and often ambiguous. For example, a demonstrative like "this" may introduce a new entity or point back to an earlier one. Eisenstein and Davis test whether a speaker's hand gestures can help decide which noun phrases refer to the same thing.

They use a small corpus of 16 coreference-annotated dialogues in which speakers explain a diagram. Hand positions are extracted from the video with computer vision. From these, two simple features are derived: the distance between the "focus hand" positions at the midpoints of two mentions, and whether the focus hand is identical across the two mentions. These features supplement standard textual features in two models. One is a pipeline that uses boosted shallow decision trees (AdaBoost) followed by average-link clustering. The other is a joint voted perceptron in the style of McCallum and Wellner.

Adding gesture significantly improves the AdaBoost system under mention-based CEAF. This suggests that non-verbal cues can support language understanding in multimodal settings such as lectures and meetings.

Key points

  • Evaluation used leave-one-out cross-validation with 16 folds (one per document), scored with mention-based CEAF F1.
  • AdaBoost reached 54.9 F1 with gesture and speech features versus 52.8 with speech alone, a significant gain (t(15) = 2.48, p < .02).
  • The voted perceptron rose only from 52.9 to 53.7, which was not significant (p = .15). Its results also varied with the order in which documents were presented.
  • All systems beat the baselines: EXACT MATCH scored 50.2, "none corefer" 41.5, and "all corefer" 18.8.
  • In a χ² analysis, the binned FOCUS DIST feature ranked fifth (χ² = 727.8). WHICH HAND was weaker but still significant (χ² = 57.2). Coreferent pairs were more likely to involve gesturing with the same hand (59.9% vs. 52.8% for non-coreferent pairs).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

8 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 7 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{eisenstein2006gesture,
  title = {{Gesture Improves Coreference Resolution}},
  author = {Eisenstein, Jacob and Davis, Randall},
  booktitle = {Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers},
  year = {2006},
  doi = {10.3115/1614049.1614059},
  url = {https://doi.org/10.3115/1614049.1614059}
}

References

13 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Breck Baldwin and Thomas Morton. 1998. Dynamic coreference-based summarization. In Proc. of EMNLP. find our card
  2. Lei Chen, Yang Liu, Mary P. Harper, and Elizabeth Shriberg. 2004. Multimodal model integration for sentence unit detection. In Proceedings of International Conference on Multimodal Interfaces (ICMI’04). ACM Press. find
  3. Michael Collins. 2002. Discriminative training methods for hidden markov models: Theory and experiments with perceptron algorithms. In Proceedings of EMNLP. find our card
  4. Erik D. Demaine and Nicole Immorlica. to appear. Correlation clustering in general weighted graphs. Theoretical Computer Science. find
  5. Jacob Eisenstein and Randall Davis. 2006. Gesture features for coreference resolution. In Workshop on Multimodal Interaction and Related Machine Learning Algorithms. find
  6. Usama M. Fayyad and Keki B. Irani. 1993. Multi-interval discretization of continuousvalued attributes for classification learning. In Proceedings of IJCAI-93. find
  7. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proc. of HLT-EMNLP, pages 25–32. find our card
  8. Andrew McCallum and Ben Wellner. 2004. Conditional models of identity uncertainty with application to noun coreference. In Neural Information Processing Systems. find
  9. Yukiko Nakano, Gabe Reinstein, Tom Stocky, and Justine Cassell. 2003. Towards a model of face-to-face grounding. In Proceedings of ACL’03. find
  10. Sharon L. Oviatt. 1999. Mutual disambiguation of recognition errors in a multimodel architecture. In Human Factors in Computing Systems (CHI’99), pages 576–583. find
  11. Elizabeth Shriberg, Andreas Stolcke, Dilek Hakkani-Tur, and Gokhan Tur. 2000. Prosody-based automatic segmentation of speech into sentences and topics. Speech Communication, 32. find
  12. Michael Strube and Christoph Müller. 2003. A machine learning approach to pronoun resolution in spoken dialogue. In Proceedings of ACL ’03, pages 168–175. find our card
  13. Ian H. Witten and Eibe Frank. 1999. Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations. Morgan Kaufmann. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/eisenstein-2006-gesture-improves-coreference-resolution/