Elamur Research⌕
☰
home / papers / 2020 / TV-AfD: An Imperative-Annotated Corpus from The Big Bang Theory and Wikipedia's Articles for Deletion Discussions

TV-AfD: An Imperative-Annotated Corpus from The Big Bang Theory and Wikipedia's Articles for Deletion Discussions

Paper · 2020 · Yimin Xiao, Zong-Ying Slaton and Lu Xiao

Summary

The paper addresses the shortage of labeled imperative data, especially from spoken conversation, which limits how well systems such as voice assistants recognise orders and requests phrased in unexpected ways. The authors built TV-AfD, a corpus with two parts:

  • Spoken-style dialogue from The Big Bang Theory (59 episodes, 25,076 statements), annotated entirely by hand.
  • Written comments from Wikipedia's Articles for Deletion (AfD) discussions. Here a syntax-based rule classifier first proposed 10,624 candidate statements, and humans then kept only the true imperatives.

The annotation scheme follows Condoravdi and Lauer (2012). It treats imperatives as a pragmatic category that goes beyond plain commands. The authors used the corpus to test the rule classifier. It does reasonably on written text (precision 0.80) but poorly on dialogue (precision 0.44, recall 0.41, F1 0.42). Error analysis shows that spoken imperatives vary more in form and depend heavily on context. Purely syntactic detection is therefore insufficient for speech. The corpus is public and could support both rule-based and supervised models.

Key points

  • Data sources: 59 episodes of The Big Bang Theory (25,076 statements) plus 10,624 rule-extracted AfD statements. The corpus is released on GitHub as TV-AfD.
  • Annotation scheme: four imperative groups, namely directives, wish-type uses, permissions/invitations/suggestions, and disinterested advice. Each group is split into negation-free and negation-included subtypes (negative words such as not, never, nobody). The finer labelling was applied only to the TV data.
  • Rule classifier: builds on Mao et al. (2014), who reached precision 0.84, recall 0.73 and F 0.79 on AfD data. It has two rules:
    • a base-form verb at the dependency root with no subject child;
    • a modal verb whose subject is a personal pronoun or noun.
  • False positives:
    • About 5% in both datasets come from questions opening with "Do you/we" or "Have you/we".
    • Pragmatic mismatches (imperative form without imperative function) cause 30% of false positives in speech but only 2% in text.
  • Other failures:
    • Tagging/parsing errors, such as unparsed "n't" or verbs tagged as adverbs, cause 20% of text false positives and 20% of speech false negatives.
    • In speech, 80% of false negatives come from rules that are too narrow to cover polite forms like "Could you get that".

Interesting ideas

  • I1. Define imperatives by what they do (wishes, offers, curses, advice) rather than only by surface syntax, so form and function are labelled separately.
  • I2. Use a cheap high-recall rule to pre-filter a large corpus, then have humans confirm the candidates; this saves annotation effort and directly measures the rule's precision.
  • I3. Split each category by whether a negative marker is present, since negated imperatives are a known blind spot for detectors.
  • I4. Break errors down by cause (parser fault, overly narrow rule, pragmatic mismatch, question look-alikes) and compare across written and spoken domains.
  • I5. Treat sitcom dialogue as a stand-in for conversational speech when real transcripts are scarce.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): prag's speech-act layer could tag both the grammatical mood and the illocutionary function, letting "Why don't you buy a better printer?" register as a suggestion despite being a question.
  • I2 (learn): silver-data pipelines could let a rule propose candidates and an LLM teacher tier or human check them, giving a precision estimate for each self-induced rule.
  • I3 (tokens): ensure the tokenizer/tagger reliably splits and tags "n't", since misparsed negation caused a fifth of the errors in both domains.
  • I4 (tree): applying the same breakdown to the UD-based imperative detector would show whether a failure belongs to gparse/the perceptron parser or to the rule on top of the tree.
  • I5 (answer): dialogue-heavy tales and fables could be evaluated on directive and request detection using a TV-AfD-style held-out set.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Syracuse UniversitySyracuse · United States3 works · 155 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

Cite

Cite as Yimin Xiao, Zong-Ying Slaton and Lu Xiao. 2020. TV-AfD: An Imperative-Annotated Corpus from The Big Bang Theory and Wikipedia's Articles for Deletion Discussions. Proceedings of the Twelfth Language Resources and Evaluation Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{xiao2020tv,
  title = {{TV-AfD: An Imperative-Annotated Corpus from The Big Bang Theory and Wikipedia's Articles for Deletion Discussions}},
  author = {Yimin Xiao, Zong-Ying Slaton and Lu Xiao},
  booktitle = {Proceedings of the Twelfth Language Resources and Evaluation Conference},
  year = {2020},
  url = {https://aclanthology.org/2020.lrec-1.805/}
}

References

15 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bies, A. et al. (2012). English Web Treebank LDC2012T13. Web Download. Philadelphia: Linguistic Data Consortium. find
  2. Cohen, W.W., Carvalho, V.R. and Mitchell, T.M. (2004). Learning to classify email into “speech acts”. Paper presented at the Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing. our card
  3. Condoravdi, C. and Lauer, S. (2012). Imperatives: Meaning and illocutionary force. Empirical issues in syntax and semantics, 9, 37-58. find
  4. Ervin-Tripp, S. (1976). Is Sybil there? The structure of some American English directives. Language in society, 5(1), 25-66. find
  5. Gupta, A., Khosla, A., Singh, G. and Dasgupta, G. (2018). Mining Procedures from Technical Support Documents. arXiv preprint arXiv:1805.09780. ↗
  6. Manning, C.D., Surdeanu, M., Bauer, J., Finkel, J. Bethard, S.J. and McClosky D. (2014). The Stanford CoreNLP natural language processing toolkit. Paper presented at the Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations. our card
  7. Mao, F., Mercer, R.E. and Xiao, L. (2014). Extracting imperatives from wikipedia article for deletion discussions. Paper presented at the Proceedings of the First Workshop on Argumentation Mining. ↗ our card
  8. Javanmardi, A. and Xiao, L. (2019). What’s in the Content of Wikipedia’s Article for Deletion Discussions? Companion Proceedings of The 2019 World Wide Web Conference on - WWW ’19, 1215–1223. https://doi.org/10.1145/3308560.3316750 ↗
  9. Kwong, H. and Yorke-Smith, N. (2012). Detection of imperative and declarative question-answer pairs in email conversations. Paper presented at the Twenty-First International Joint Conference on Artificial Intelligence. ↗
  10. McHugh, M. L. (2012). Interrater reliability: the kappa statistic. Biochemia medica, 22(3), 276–282. Minkov, E., Wang, R.C. and Cohen, W.W. (205). Extracting personal names from email: Applying named entity recognition to informal text. Paper presented at the Proceedings of human language technology conference and conference on empirical methods in natural language processing. ↗
  11. Searle, J. R. (1976). A classification of illocutionary acts. Language in society, 5(1), 1-23. ↗
  12. Shrestha, L. and Mckeown, K. (2004). Detection of question-answer pairs in email conversations. Paper presented at the Proceedings of the 20th international conference on Computational Linguistics. find
  13. Quaglio, P. (2009). Television Dialogue: The Sitcom Friends Vs. Natural Conversation. Amsterdam: John Benjamins Publishing Co. The Big Bang Theory. Retrieved from https://en.wikipedia.org/wiki/The_Big_Bang_Theory#c ite_note-3 ↗
  14. Thompson, C. M., Crook, B., Love, B., Macpherson, C. F., & Johnson, R. (2016). Understanding how adolescents and young adults with cancer talk about needs in online and face-to-face support groups. Journal of Health Psychology, 21(11), 2636-2646. ↗
  15. Viera, A. J., & Garrett, J. M. (2005). Understanding Interobserver Agreement: The Kappa Statistic. Family Medicine, 37(5), 360-363. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xiao-2020-tv-afd-imperative-annotated-corpus/