Elamur Research⌕
☰
home / papers / 2022 / Dialog Acts for Task Driven Embodied Agents

Dialog Acts for Task Driven Embodied Agents

Paper · 2022 · Spandana Gella, Aishwarya Padmakumar, Patrick Lange et al.

Summary

Household robots that take instructions in natural language must understand requests given at different levels of detail and know when to ask for missing information. Dialog act schemes from conventional task-oriented dialog assume that tasks reduce to slot-value pairs, which fits poorly with situated conversations. The authors build a new taxonomy of 18 dialog acts in five groups, derived from what speakers actually do in TEACh. TEACh is a corpus of more than 3,000 free-form text chats in which a Commander guides a Follower through household chores in simulation. Labelling all of it produces TEACh-DA, one of the first large dialog act resources for embodied task completion. They then use it in four settings: tagging the act of an utterance, forecasting the act of the next turn, predicting object-manipulation plans, and Execution from Dialog History (EDH). Forecasting the next act proves hard because the chats have no fixed turn taking. Inserting dialog act markers into the language input of the Episodic Transformer raises EDH success rate by as much as 2 points. The work shows that communicative intent can feed into physical action, not only into dialog management.

Key points

  • TEACh-DA has 39.5k utterances from about 3,000 dialogs. Roughly 60% of the turns come from the Commander and the remainder from the Follower.
  • The 18 acts fall into five groups:
    • Generic (e.g., Greetings, Acknowledge, Affirm, Deny, Confirm)
    • Instruction (e.g., Instruction, ReqForInstruction, RequestMore)
    • Object/Location (e.g., InfoObjectLocAndOD, ReqForObjLocAndOD, AlternateQuestions)
    • Feedback
    • Interface (e.g., NotifyFailure)
  • The most frequent labels are Instruction (11,019), Acknowledge (7,421) and InfoObjectLocAndOD (6,946). Speaker roles split sharply by act: Commanders produce almost all instructions, and Followers produce almost all requests for instructions.
  • A single utterance may carry several acts. Annotators mark the text span that belongs to each act, following the DAMSL-style practice of Core and Allen (1997).
  • Plan prediction is scored with three measures: the share of well-formed plans, and precision and recall of (action, object) pairs. On EDH, the best variants add dialog act tags at the end (+DA-E) or at both ends (+DA-SE) of each utterance. Speaker tags (+ST) gave little benefit.

Interesting ideas

  • I1. Instead of a generic Inform act with one slot that swallows most of the sentence, the schema uses fine-grained, domain-specific acts such as object-location information. These are designed so they can later be merged back into broad Inform/Request plus intent or slot labels.
  • I2. Multiple acts per utterance, each anchored to its own text span, handle turns that do several things at once, such as acknowledging and then asking.
  • I3. Speaker-role statistics per act (who issues instructions and who asks for them) act as a cheap consistency signal about each participant's role.
  • I4. Simply wrapping each utterance with its dialog act tag, at the end or at both ends, helped an end-to-end action model, while speaker tags did not. This suggests that explicit intent markup carries signal a model does not reliably recover alone.
  • I5. Plan outputs are checked structurally (valid alternation of action and object) before precision and recall of (action, object) pairs are measured. Ill-formed outputs are therefore caught separately from wrong content.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): The speech-act layer in prag could use detailed act types that map down to coarse classes, so explanations stay specific while compatibility with standard tagsets is kept.
  • I2 (frames): Speech acts could be attached to clause or span nodes in the UD tree rather than to whole sentences, which lets a single sentence carry both a confirmation and a question.
  • I3 (world): The world model could record speaker roles in dialogue-like texts such as fables, and use act-role patterns to infer who holds authority or knowledge.
  • I4 (answer): Readers that solve word problems or tale QA could receive act-annotated sentences (instruction versus information versus question), helping them separate the facts given from what is being asked.
  • I5 (back): The text → graph → text compressor could first verify that the generated graph or plan is well formed, then score content tuples against the reference, keeping both error types explainable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksSocial Psychology · 306 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 624 worksSpeech act theory · 229 works

Cite

Cite as Spandana Gella, Aishwarya Padmakumar, Patrick Lange and Dilek Hakkani-Tur. 2022. Dialog Acts for Task Driven Embodied Agents. Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.sigdial-1.13 · arXiv 2209.12953

@inproceedings{gella2022dialog,
  title = {{Dialog Acts for Task Driven Embodied Agents}},
  author = {Spandana Gella, Aishwarya Padmakumar, Patrick Lange and Dilek Hakkani-Tur},
  booktitle = {Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue},
  year = {2022},
  doi = {10.18653/v1/2022.sigdial-1.13},
  eprint = {2209.12953},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2022.sigdial-1.13}
}

References

27 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Cristian-Paul Bara, CH-Wang Sky, and Joyce Chai. 2021. Mindcraft: Theory of mind modeling for situated dialogue in collaborative tasks. In Proceedings of EMNLP 2021, pages 1112–1125. find
  2. Valts Blukis, Chris Paxton, Dieter Fox, Animesh Garg, and Yoav Artzi. 2022. A persistent spatial semantic representation for high-level natural language in-Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In EMNLP.
  3. Harry Bunt, Dirk K. J. Heylen, Catherine Pelachaud, Roberta Catizone, and David R. Traum. 2009. The dit++ taxanomy for functional dialogue markup. In EDAML@AAMAS, Workshop Towards a Standard Markup Language for Embodied Dialogue Acts, pages 13–24. find
  4. Bill Byrne, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Ben Goodrich, Daniel Duckworth, Semih Yavuz, Amit Dubey, Kyu-Young Kim, and Andy Cedilnik. 2019. Taskmaster-1: Toward a realistic and diverse dialog dataset. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4516–4525. ↗
  5. Ta-Chung Chi, Minmin Shen, Mihail Eric, Seokhwan Kim, and Dilek Hakkani-Tür. 2020. Just ask: An interactive learning framework for vision and language navigation. In AAAI 2020, pages 2459–2466. AAAI Press. find
  6. Mark G. Core and James F. Allen. 1997. Coding dialogs with the damsl annotation scheme. In Working Notes of the AAAI Fall Symposium on Communicative Action in Humans and Machines, pages 28–35. find
  7. Song Feng, Hui Wan, Chulaka Gunasekara, Siva Patel, Sachindra Joshi, and Luis Lastras. 2020. doc2dial: A goal-oriented document-grounded dialogue dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8118–8128.
  8. Felix Gervits, Antonio Roque, Gordon Briggs, Matthias Scheutz, and Matthew Marge. 2021. How should agents ask questions for situated learning? an annotated dialogue corpus. In Proceedings of the SIGDIAL 2021, pages 353–359. ↗
  9. Felix Gervits, Ravenna Thielstrom, Antonio Roque, and Matthias Scheutz. 2020. It’s about time: Turn-entry timing for situated human-robot dialogue. In Proceedings of the SIGDIAL 2020, pages 86–96. find
  10. Malik Ghallab, Dana Nau, and Paolo Traverso. 2016. Automated planning and acting. Cambridge University Press. ↗
  11. Charles T Hemphill, John J Godfrey, and George R Doddington. 1990. The atis spoken language systems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990. find
  12. Zhiwei Jia, Kaixiang Lin, Yizhou Zhao, Qiaozi Gao, Govind Thattai, and Gaurav Sukhatme. 2022. Learning to act with affordance-aware multimodal neural slam. arXiv preprint arXiv:2201.09862. ↗
  13. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent convolutional neural networks for discourse compositionality. In Proceedings of the Workshop on Continuous Vector Space Models and their Compositionality, pages 119–126.
  14. Simon Keizer, Milica Gasic, Filip Jurcicek, François Mairesse, Blaise Thomson, Kai Yu, and Steve Young. 2010. Parameter estimation for agenda-based user simulation. In Proceedings of the SIG-DIAL 2010 Conference, pages 116–123. find
  15. Satwik Kottur, Seungwhan Moon, Alborz Geramifard, and Babak Damavandi. 2021. SIMMC 2.0: A task-oriented dialog dataset for immersive multimodal conversations. In Proceedings of EMNLP, pages 4903–4912.
  16. Amrita Saha, Mitesh Khapra, and Karthik Sankaranarayanan. 2018. Towards building large scale multimodal domain-aware conversation systems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32. ↗
  17. Jost Schatzmann and Steve Young. 2009. The hidden agenda user simulation model. IEEE transactions on audio, speech, and language processing, 17(4):733–747. ↗
  18. Sebastian Schuster, Sonal Gupta, Rushin Shah, and Mike Lewis. 2019. Cross-lingual transfer learning for multilingual task oriented dialog. In Proceedings of NAACL-HLT, pages 3795–3805. ↗
  19. Ayush Shrivastava, Karthik Gopalakrishnan, Yang Liu, Robinson Piramuthu, Gökhan Tür, Devi Parikh, and Dilek Hakkani-Tür. 2021. VISITRON: visual semantics-aligned interactively trained object-navigator. CoRR, abs/2105.11589. ↗
  20. Ji Young Lee and Franck Dernoncourt. 2016. Sequential short-text classification with recurrent and convolutional neural networks. In NAACL HLT, pages 515–520.
  21. Koji Tanaka, Junya Takayama, and Yuki Arase. 2019. Dialogue-act prediction of future responses based on conversation history. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL, pages 197–202.
  22. Stefano Mezza, Alessandra Cervone, Evgeny A. Stepanov, Giuliano Tortoreto, and Giuseppe Riccardi. 2018. Iso-standard domain-independent dialogue act tagging for conversational agents. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3539–3551. ↗
  23. Jason D Williams, Matthew Henderson, Antoine Raux, Blaise Thomson, Alan Black, and Deepak Ramachandran. 2014. The dialog state tracking challenge series. AI Magazine, 35(4):121–124. ↗
  24. So Yeon Min, Devendra Singh Chaplot, Pradeep Ravikumar, Yonatan Bisk, and Ruslan Salakhutdinov. 2021. Film: Following instructions in language with modular methods. arXiv preprint arXiv:2110.07342. ↗
  25. Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan-Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur. 2021. Teach: Task-driven embodied agents that chat. arXiv preprint arXiv:2110.00534. ↗
  26. Steve Young. 2007. Cued standard dialogue acts. find
  27. Report, Cambridge University Engineering Department, 2007. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gella-2022-dialog-acts-task-driven-embodied/