Elamur Research⌕
☰
home / papers / 2018 / A Supervised Approach To The Interpretation Of Imperative To-Do Lists

A Supervised Approach To The Interpretation Of Imperative To-Do Lists

Paper · 2018 · Paul Landes and Barbara Di Eugenio

Summary

The paper studies automatic interpretation of personal to-do items. These are very short entries, written either as bare imperatives ("Call mom") or as verbless fragments ("Hotel reservation"). Each item is mapped to an intelligent agent (IA) type that could carry it out, such as buy, call, email or calendar, and its typed arguments are extracted (item, person, holiday and so on). Short telegraphic input breaks standard tools, so the authors adapt the Stanford pipeline. They add NER fed with compiled entity lists and a dedicated model that finds the first verb, repairing POS tagger mistakes. Classifiers in Weka then use features from word counts, named entities, dependencies, PropBank role labels and word2vec similarity. Argument tagging is done with a separate model per IA. LogitBoost performs best for IA classification and shows a similar pattern on a second, publicly released corpus. Argument F1 ranges from roughly 0.53 to 0.88 depending on the IA. The work matters because imperatives and fragments are rare in standard corpora, yet they are the main way users state tasks to assistants.

Key points

  • Corpus A (proprietary) has 3,169 annotated tasks. Of these, 1,479 were set aside as exceptions: ambiguous, non-English, meaningless, illegal or generic-professional items. After pruning IAs with fewer than 15 examples, 1,611 tasks remain.
  • IA annotation agreement reached a Fleiss' kappa of 0.679 between two annotators. Buy is the dominant IA at 28.5% of annotations.
  • Corpus B combines 102 volunteer tasks and 498 Trello tasks, with 286 used in experiments. It is released at github.com/plandes/todo-task and carries IA labels only, without arguments.
  • A word-vector cosine-similarity feature, weighted by per-IA word frequency, added about 5 F-measure points to IA classification.
  • On Corpus B, LogitBoost reaches F1 0.71. Per-IA argument F1 on Corpus A runs from about 0.53 (school-work) up to 0.88.

Interesting ideas

  • I1. Items that cannot be labelled reliably are moved into an explicit exception class instead of being forced into a category, which keeps the training labels clean and makes the system's refusals visible.
  • I2. A small dedicated model locates the main verb before parsing, because general POS taggers often mislabel the leading verb in short imperatives.
  • I3. Argument labels are predicted by a separate model chosen by the predicted intent, so each intent has its own slot inventory.
  • I4. Verbless fragments like "Hotel reservation" are understood to imply a missing head action, here schedule.
  • I5. Adjacent tokens tagged with the same argument type are merged into a single span, for example "blue sweater" becoming one item.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver data could gain a documented "exception" bucket (ambiguous, foreign, junk), which would also serve as a ready negative control during evaluation.
  • I2 (tokens): A targeted first-verb correction step for imperative and headless inputs could stop tagger errors from spreading into the UD tree and frames.
  • I3 (frames): Role assignment could be conditioned on the already chosen VerbNet class or speech act, which narrows the candidate roles and makes each choice easier to explain.
  • I4 (frames): A rule layer could rebuild an implicit predicate for nominal fragments, so that a full who-does-what frame still enters the world of the text.
  • I5 (world): The same merge rule could turn token-level role tags into single entity mentions before they are stored as world objects.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Illinois ChicagoChicago · United States38 works · 19,844 citations

📚 Shelves 7 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Information Systems · 523 works Decision Sciences · 85 works Information Systems and Management · 18 works

Cite

Cite as Paul Landes and Barbara Di Eugenio. 2018. A Supervised Approach To The Interpretation Of Imperative To-Do Lists. arXiv:1806.07999.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1806.07999 · arXiv 1806.07999

@misc{landes2018supervised,
  title = {{A Supervised Approach To The Interpretation Of Imperative To-Do Lists}},
  author = {Paul Landes and Barbara {Di Eugenio}},
  year = {2018},
  doi = {10.48550/arxiv.1806.07999},
  eprint = {1806.07999},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1806.07999}
}

References

33 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Victoria Bellotti, Brinda Dalal, Nathaniel Good, Peter Flynn, Daniel G Bobrow, and Nicolas Ducheneaut. 2004. What a to-do: studies of task management towards the design of a personal task list manager. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 735–742. ACM. find
  2. Satchuthananthavale RK Branavan, Harr Chen, Luke S Zettlemoyer, and Regina Barzilay. 2009. Reinforcement learning for mapping instructions to actions. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1-Volume 1, pages 82–90. Association for Computational Linguistics. find
  3. Angel X Chang and Christopher D Manning. 2014. Tokensregex: Defining cascaded regular expressions over tokens. Technical report, Technical Report CSTR 2014-02, Department of Computer Science, Stanford University. find
  4. Jinho D Choi. 2014. Optimization of natural language processing components for robustness and scalability. Ph.D. thesis, University of Colorado Boulder. find
  5. Open Product Data. 2016. Open product data. http://product-open-data.com. ↗
  6. Marie-Catherine De Marneffe, Bill MacCartney, Christopher D Manning, et al. 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of LREC, volume 6, pages 449–454. our card
  7. Anthony Fader, Stephen Soderland, and Oren Etzioni. 2011. Identifying relations for open information extraction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1535–1545. Association for Computational Linguistics. find
  8. Jenny Rose Finkel, Trond Grenager, and Christopher Manning. 2005. Incorporating non-local information into information extraction systems by gibbs sampling. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pages 363–370. Association for Computational Linguistics. our card
  9. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. ↗
  10. Eibe Frank and Ian H. Witten. 1998. Generating accurate rule sets without global optimization. In Fifteenth International Conference on Machine Learning, pages 144–151. Morgan Kaufmann. find
  11. J. Friedman, T. Hastie, and R. Tibshirani. 1998. Additive logistic regression: a statistical view of boosting. Technical Report Vol. 28, No. 2, The Annals of Statistics, Stanford University. find
  12. Rayid Ghani, Katharina Probst, Yan Liu, Marko Krema, and Andrew Fano. 2006. Text mining for product attribute extraction. ACM SIGKDD Explorations Newsletter, 8(1):41–48. ↗
  13. Yolanda Gil, Varun Ratnakar, Timothy Chklovski, Paul Groth, and Denny Vrandecic. 2012. Capturing common knowledge about tasks: Intelligent assistance for to-do lists. ACM Transactions on Interactive Intelligent Systems (TiiS), 2(3):15. ↗
  14. Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H Witten. 2009. The weka data mining software: an update. ACM SIGKDD explorations newsletter, 11(1):10–18. ↗
  15. Bo Han and Timothy Baldwin. 2011. Lexical normalisation of short text messages: Makn sens a# twitter. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 368–378. Association for Computational Linguistics. our card
  16. Jiepu Jiang, Ahmed Hassan Awadallah, Rosie Jones, Umut Ozertem, Imed Zitouni, Ranjitha Gurunath Kulkarni, and Omar Zia Khan. 2015. Automatic Online Evaluation of Intelligent Assistants. In Proceedings of the 24th International Conference on World Wide Web, WWW ’15, pages 506–516, Republic and Canton of Geneva, Switzerland. International World Wide Web Conferences Steering Committee. find
  17. Stephen R Jones and Peter J Thomas. 1997. Empirical assessment of individuals’ personal information management systems. Behaviour & Information Technology, 16(3):158–160. ↗
  18. Ruth Kempson, Ronnie Cann, Eleni Gregoromichelaki Arasheshghi, and Matthew Purver. 2015. Ellipsis. In The Handbook of Contemporary Semantic Theory, chapter 4. John Wiley & Sons. find
  19. Tracy Holloway King, Richard Crouch, Stefan Riezler, Mary Dalrymple, and Ronald M. Kaplan. 2003. The PARC 700 dependency bank. In Proceedings of 4th International Workshop on Linguistically Interpreted Corpora (LINC-03) at EACL 2003. find
  20. Paul Kingsbury and Martha Palmer. 2002. From treebank to propbank. In LREC, pages 1989– 1993. ↗ our card
  21. Andrew Ko. 2015. Word2vecJava. https://github.com/medallia/Word2VecJava. ↗
  22. Stephen C Levinson. 1983. Pragmatics (cambridge textbooks in linguistics). find
  23. Matt Mahoney. 2006. Rationale for a large text compression benchmark. find
  24. Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. 1994. The penn treebank: annotating predicate argument structure. In Proceedings of the workshop on Human Language Technology, pages 114–119. Association for Computational Linguistics. our card
  25. Philipp Meerkamp and Zhengyi Zhou. 2017. Boosting information extraction systems with character-level neural networks and free noisy supervision. In Proceedings of the 2nd Workshop on Structured Prediction for Natural Language Processing, pages 44–51. find
  26. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. ↗ our card
  27. George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41. ↗ our card
  28. T. Mitchell, W. Cohen, E. Hruschka, P. Talukdar, J. Betteridge, A. Carlson, B. Dalvi, M. Gardner, B. Kisiel, J. Krishnamurthy, N. Lao, K. Mazaitis, T. Mohamed, N. Nakashole, E. Platanios, A. Ritter, M. Samadi, B. Settles, R. Wang, D. Wijaya, A. Gupta, X. Chen, A. Saparov, M. Greaves, and J. Welling. 2015. Never-ending learning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI-15). ↗
  29. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: An annotated corpus of semantic roles. Computational linguistics, 31(1):71–106. ↗ our card
  30. Eric Prudhommeaux and Andy Seaborne. 2006. Sparql query language for rdf. w3c working draft, 4 october 2006. SPARQL Query Language for RDF. find
  31. Duangmanee Pew Putthividhya and Junling Hu. 2011. Bootstrapped named entity recognition for product attribute extraction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1557–1567. Association for Computational Linguistics. find
  32. Gokhan Tur, Anoop Deoras, and Dilek Hakkani-Tür. 2014. Detecting out-of-domain utterances addressed to a virtual personal assistant. In Fifteenth Annual Conference of the International Speech Communication Association. find
  33. Wikidata. 2015. Q5488768 — wikidata. http://www.wikidata.org/w/index.php?title=Q5488768&oldid=8079009. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/landes-2018-supervised-approach-interpretation-imperative-lists/