Elamur Research⌕
☰
home / papers / 2020 / TED-Q: TED Talks and the Questions they Evoke

TED-Q: TED Talks and the Questions they Evoke

Paper · 2020 · Matthijs Westera, Laia Mayol and Hannah Rohde

Summary

Most of the questions that organise a discourse, the Questions Under Discussion (QUDs), are never stated outright. That makes them hard to recover, and expert annotation of discourse relations is expensive and tied to particular theories. The authors propose a forward-looking, two-step crowdsourcing task. Non-expert readers see a text in small increments and type the question the passage so far makes them wonder about. They are then shown the continuation and judge whether their question was answered. The method is applied to the English portion of TED-MDB, a set of scripted TED talks that already carry PDTB-style discourse relation annotations, and produces a new annotation layer called TED-Q.

The proportion of evoked questions that later get answered serves as an estimate of how predictable the QUD is at each point. Comparing the two layers shows that connectives are omitted more often where the QUD was more predictable. This supports predictability-driven reduction (the Uniform Information Density idea) at the level of discourse. The data, interfaces and scripts are public, and the procedure is designed to scale for NLP uses such as dialogue systems and conversational QA.

Key points

  • The base corpus is TED-MDB (Zeyrek et al., 2018), English part only: six scripted TED talks, 6975 words, with PDTB-style discourse relations.
  • The annotation runs in two steps:
    • Annotators write a free-form evoked question while the following text is still hidden.
    • After seeing the continuation, they judge whether it answered that question.
  • A pilot on the unscripted DISCO-SPICE corpus was dropped because participants found its transcripts too hard to follow.
  • The main empirical result is that explicit marking of discourse relations is less frequent at positions where the QUD was more predictable.
  • TED-Q is released as a preliminary first installment, together with the annotation interfaces and analysis scripts, at github.com/amore-upf/ted-q.

Interesting ideas

  • I1. Hiding the continuation until a question is posed captures what readers expect next, and the rate at which those expectations are answered becomes a cheap, numeric measure of predictability.
  • I2. Asking for a free-form question instead of a label from a large relation inventory makes the task natural for non-experts, and questions still map onto relations (a "Why?" question answered by the next clause corresponds to a causal link).
  • I3. Adding a new layer on top of a corpus that is already annotated lets a hypothesis be tested by correlating the two independent layers, instead of collecting everything from scratch.
  • I4. The link between high predictability and omitted connectives suggests a production rule: leave a connective out when the relation it signals is already strongly expected.
  • I5. The pattern of asking a question mid-text and later checking whether it was answered is itself a reading-comprehension routine.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): at each clause boundary, prag could generate its own expected questions (why? what happened next?) and check whether the next clause fills them, giving an explainable signal for implicit discourse relations.
  • I2 (learn): LLM teacher tiers could be asked for evoked questions rather than relation labels, with a deterministic mapping from question type to relation frame, so teacher output is easier to check and less theory-biased.
  • I3 (learn): Mova's predicted implicit relations could be scored against TED-MDB's PDTB layer, with explicit-connective cases acting as a control.
  • I4 (back): the generator could drop "because" or "then" when the meaning graph makes the evoked question highly predictable, and keep them when it does not, producing more natural text.
  • I5 (answer): readers of fables and tales could keep a list of their own open questions while reading and report which ones the story resolved, giving traceable evidence of understanding.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Literature and Literary Theory · 340 works

🧭 Topics 1 Discourse analysis · 394 works

Cite

Cite as Matthijs Westera, Laia Mayol and Hannah Rohde. 2020. TED-Q: TED Talks and the Questions they Evoke. Proceedings of the Twelfth Language Resources and Evaluation Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{westera2020ted,
  title = {{TED-Q: TED Talks and the Questions they Evoke}},
  author = {Matthijs Westera, Laia Mayol and Hannah Rohde},
  booktitle = {Proceedings of the Twelfth Language Resources and Evaluation Conference},
  year = {2020},
  url = {https://aclanthology.org/2020.lrec-1.141/}
}

References

50 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthonio, T. R., Bhat, I. A., and Roth, M. (2020). wikihowtoimprove: A resource and analyses on edits in instructional texts. In Proceedings of the Twelfth International Conference on Language Resources and Evaluation (LREC’2020), Marseille, France, May. European Language Resource Association (ELRA). find
  2. Arora, S., Liang, Y., and Ma, T. (2017). A simple but tough-to-beat baseline for sentence embeddings. In International Conference on Learning Representations. find
  3. Asher, N. and Lascarides, A. (2003). Logics of Conversation. Cambridge University Press, Cambridge. find
  4. Asr, F. T. and Demberg, V. (2012). Implicitness of discourse relations. In Proceedings of COLING 2012, pages 2669–2684. our card
  5. Aylett, M. and Turk, A. (2004). The smooth signal redundancy hypothesis: A functional explanation for relationships between redundancy, prosodic prominence, and duration in spontaneous speech. Language and Speech, 47:31–56. ↗
  6. Beaver, D. I. and Clark, B. Z. (2008). Sense and Sensitivity: How Focus Determines Meaning. Wiley-Blackwell, West Sussex, UK. find
  7. Bowman, S. R., Angeli, G., Potts, C., and Manning, C. D. (2015). A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326. ↗
  8. Carlson, L. (1983). Dialogue Games: An Approach to Discourse Analysis. Reidel, Dordrecht. find
  9. Craggs, R. and Wood, M. M. (2005). Evaluating discourse and dialogue coding schemes. Computational Linguistics, 31(3):289–296. ↗
  10. Cummins, C. and Rohde, H. (2015). Evoking context with contrastive stress: Effects on pragmatic enrichment. Frontiers in Psychology, Special issue on Context in communication: A cognitive view, 6:1–11. ↗
  11. De Kuthy, K., Reiter, N., and Riester, A. (2018). Qud-based annotation of discourse structure and information structure: Tool and evaluation. In Nicoletta Calzolari et al., editor, Proceedings of the 11th Language Resources and Evaluation Conference (LREC), pages 1932–1938. our card
  12. Frank, A. F. and Jaeger, T. F. (2008). Speaking rationally: Uniform information density as an optimal strategy for language production. In Proceedings of the Annual Meeting of the Cognitive Science Society. find
  13. Gahl, S. (2008). “time” and “thyme” are not homophones: Word durations in spontaneous speech. Language, 84:474–496. find
  14. Ginzburg, J. and Sag, I. (2000). Interrogative Investigations. CSLI Publications, Stanford. find
  15. Ginzburg, J. (1994). An update semantics for dialogue. In Proceedings of the Tilburg International Workshop on Computational Semantics. find
  16. Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters. Advanced Analytics, LLC. find
  17. Hewett, F., Rane, R. P., Harlacher, N., and Stede, M. (2019). The utility of discourse parsing features for predicting argumentation structure. In Proceedings of the 6th Workshop on Argument Mining, pages 98–103. our card
  18. Hirschberg, J. and Litman, D. (1993). Empirical studies on the disambiguation of cue phrases. Computational Linguistics, 19:501– 530. our card
  19. Hobbs, J. R. (1979). Coherence and coreference. Cognitive Science, 3:67–90. ↗
  20. Kehler, A. and Rohde, H. (2013). A probabilistic reconciliation of coherence-driven and centering-driven theories of pronoun interpretation. Theoretical Linguistics, 39:1– 37. ↗
  21. Kehler, A. and Rohde, H. (2017). Evaluating an expectation-driven qud model of discourse interpretation. Discourse Processes, 54:219–238. find
  22. Kehler, A. (2002). Coherence, reference, and the theory of grammar. CSLI Publications, Stanford, CA. find
  23. Kim, C. S., Gunlogson, C., Tanenhaus, M. K., and Runner, J. T. (2015). Context-driven expectations about focus alternatives. Cognition, 139:28–49. ↗
  24. Knott, A. (1996). A data-driven methodology for motivating a set of coherence relations. Ph.D. thesis, University of Edinburgh. find
  25. Landis, J. R. and Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174. ↗
  26. Larsson, S. (1998). Questions under discussion and dialogue moves. In Proceedings of TWLT 13/Twendial ’98: Formal Semantics and Pragmatics of Dialogue. find
  27. Levy, R. and Jaeger, T. F. (2007). Speakers optimize information density through syntactic reduction. In Advances in Neural Information Processing Systems, page 849–856. ↗
  28. Mann, W. C. and Thompson, S. A. (1988). Rhetorical structure theory: Toward a functional theory of text organization. Text, 8:243–281. ↗ our card
  29. McDonald, S. and Ramscar, M. (2001). Testing the distributional hypothesis: The influence of context on judgements of semantic similarity. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 23. find
  30. Murray, J. D. (1997). Connectives and narrative text: The role of continuity. Memory & Cognition, 25(2):227–236. ↗
  31. Onea, E. (2016). Potential questions at the semantics-pragmatics interface. Brill. ↗
  32. Polanyi, L. (1988). A formal model of the structure of discourse. Journal of Pragmatics, 12:601–638. ↗
  33. Riester, A. (2019). Constructing qud trees. In Klaus v. Heusinger, et al., editors, Questions in Discourse, volume 2. Brill, Leiden. ↗
  34. Roberts, C. (1996). Information structure in discourse: Towards an integrated formal theory of pragmatics. OSU Working Papers in Linguistics, 49: Papers in Semantics. our card
  35. Rohde, H., Johnson, A., Schneider, N., and Webber, B. (2018). Discourse coherence: Concurrent explicit and implicit relations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL). our card
  36. Sanders, T. J., Demberg, V., Hoek, J., Scholman, M. C., Asr, F. T., Zufferey, S., and Evers-Vermeul, J. (2018). Unifying dimensions in coherence relations: How various annotation frameworks are related. Corpus Linguistics and Linguistic Theory. ↗
  37. Sanders, T. (2005). Coherence, causality and cognitive complexity in discourse. In Proceedings/Actes SEM-05, First International Symposium on the exploration and modelling of meaning, pages 105–114. University of Toulouse-le-Mirail Toulouse. find
  38. Segal, E. M., Duchan, J. F., and Scott, P. J. (1991). The role of interclausal connectives in narrative structuring: Evidence from adults’ interpretations of simple stories. Discourse processes, 14(1):27–54. ↗
  39. Siddharthan, A. (2003). Preserving discourse structure when simplifying text. In Proceedings of the European Natural Language Generation Workshop (ENLG), 11th Conference of the European Chapter of the Association for Computational Linguistics (EACL). find
  40. Turnbull, R. (2018). Patterns of probabilistic segment deletion/ reduction in english and japanese. Linguistics Vanguard, pages 1–14. ↗
  41. van Kuppevelt, J. (1995). Discourse structure, topicality, and questioning. Journal of Linguistics, 31:109–147. ↗
  42. Voita, E., Serdyukov, P., Sennrich, R., and Titov, I. (2018). Context-aware neural machine translation learns anaphora resolution. In the 56th Annual Meeting of the Association for Computational Linguistics, pages 1264– 1274. our card
  43. Westera, M. and Rohde, H. (2019). Asking between the lines: elicitation of evoked questions from text. In Proceedings of the Amsterdam Colloquium. find
  44. Wolf, F. and Gibson, E. (2005). Representing discourse coherence: A corpus-based study. Computational Linguistics, 31:249–288. ↗ our card
  45. Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016). Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. ↗
  46. Xiong, H., He, Z., Wu, H., and Wang, H. (2019). Modeling coherence for discourse neural machine translation. In the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI), pages 7338–7345. find
  47. Yung, F., Scholman, M., and Demberg, V. (2019). Crowdsourcing discourse relation annotations by a two-step connective insertion task. In Linguistic Annotation Workshop LAW. 9. ↗ our card
  48. Language Resource References Rashmi Prasad and Bonnie Webber and Alan Lee and Aravind Joshi. (2019). Penn Discourse Treebank version 3.0. Linguistic Data Consortium, 3.0, ISLRN 977-491-842-427-0. ↗
  49. Ines Rehbein and Scholman Merel and Demberg Vera. (2016). Annotating Discourse Relations in Spoken Language: A Comparison of the PDTB and CCR Frameworks. (Disco-SPICE corpus). our card
  50. Zeyrek, D. and Mendes, A. and Grishina, Y. and Kurfalı, M. and Gibbon, S. and Ogrodniczuk, M. (2018). TED multilingual discourse bank (TED-MDB): a parallel corpus annotated in the PDTB style. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/westera-2020-ted-q-ted-talks-questions/