Elamur Research⌕
☰
home / papers / 2023 / Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the CoDraw Dataset

Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the CoDraw Dataset

Paper · 2023 · Brielen Madureira and David Schlangen

Summary

The paper studies how players in instruction-following games ask for clarification when an instruction is understood but too vague or ambiguous to act on with confidence. These are Instruction Clarification Requests (iCRs), situated at Clark's level of uptake. The authors annotate iCRs in CoDraw, an existing corpus of a collaborative game where a teller describes a clipart scene and a drawer rebuilds it. The resulting resource, CoDraw-iCR (v1), contains about 8.8k iCRs across 9.9k dialogues. These requests arise spontaneously from drawers, not from prompts by the data collectors, and they vary in wording and content. A corpus analysis shows that iCR turns and the turns following them change the dynamics of the game. The authors then define two baseline tasks. One is predicting when the drawer should ask, and the other is recognising that a drawer utterance is an iCR. Recognition turns out easy even for a logistic regression model, while predicting when to ask remains hard. The work offers a large, naturally occurring resource for learning clarification behaviour from data.

Key points

  • CoDraw-iCR (v1) adds iCR labels to the CoDraw dataset (Kim et al., 2019), giving roughly 8.8k iCRs over 9.9k dialogues and making it a large spontaneous iCR corpus.
  • The authors propose six desiderata for a clarification dataset: naturalness, specificity, frequency, diversity, relevance and regularity.
  • There are two tasks, both evaluated with AP and macro-F1:
    • Task 1 (IF side) asks when to make an iCR and remains difficult. Downsampling, upsampling, cost-sensitive losses and Transformer variants did not substantially help.
    • Task 2 (IG side) detects iCRs and is solved well even by logistic regression.
  • Input ablations show that the drawer's utterances carry useful information, which argues against the "silent drawer assumption" of the original CoDraw models.
  • The dataset has several limitations:
    • Labels were assigned to utterance types rather than to each utterance in full context.
    • Crowdworker IDs are unavailable.
    • Actions in the game are reversible.
    • The teller's peek action may weaken incentives to give exhaustive instructions.
    • A second, fine-grained annotation phase is planned.

Interesting ideas

  • I1. Before collecting or adopting data for a phenomenon, the authors list explicit quality criteria (natural, specific, frequent, diverse, relevant, regular) and check the corpus against each one, which makes the choice of dataset defensible.
  • I2. Labels are assigned to distinct utterance forms instead of to every token in context, so that very frequent, repetitive utterances are judged once and annotation scales to thousands of dialogues.
  • I3. The authors separate "understood but not actionable" (uptake level) from perception or parsing failures, treating underspecification of action parameters such as size or orientation as its own trigger for asking.
  • I4. Clarification is split into two asymmetric subtasks: deciding to ask, which is hard, and recognising that a question was asked, which is easy. Each is evaluated separately so the difficulty is localized.
  • I5. Ablating input components exposed that the supposedly passive partner's turns were informative, overturning a modelling assumption.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Use a similar checklist to audit silver data and LLM-teacher output, so that synthetic or induced examples are flagged before they shape training.
  • I2 (learn): When agreeing silver labels across witness parsers, cluster identical sentence or fragment types and adjudicate each type once, saving teacher calls on repetitive inputs.
  • I3 (world): When building the world of a text, mark entity slots that a downstream step needs but the text leaves open, and surface them as explicit unknowns rather than silently filling defaults.
  • I4 (frames): Add a clarification-request speech-act label in prag that is detected from form, kept distinct from the harder decision about whether the text truly requires clarification.
  • I5 (answer): Run ablations that drop dialogue participants or discourse segments when evaluating QA on tales, to verify which context the reader actually relies on instead of assuming it.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬German Research Centre for Artificial IntelligenceKaiserslautern · Germany79 works · 29,970 citations🎓University of PotsdamPotsdam · Germany44 works · 5,305 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Vision & language · 811 worksDialogue systems · 654 works

Cite

Cite as Brielen Madureira and David Schlangen. 2023. Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the CoDraw Dataset. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.eacl-main.169

@inproceedings{madureira2023instruction,
  title = {{Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the CoDraw Dataset}},
  author = {Brielen Madureira and David Schlangen},
  booktitle = {Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.eacl-main.169},
  url = {https://doi.org/10.18653/v1/2023.eacl-main.169}
}

References

64 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeff Dalton, and Mikhail Burtsev. 2021. Building and evaluating open-domain dialogue corpora with clarifying questions. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4473–4484, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  2. Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, and W Bruce Croft. 2019. Asking clarifying questions in open-domain information-seeking conversations. In Proceedings of the 42nd international acm sigir conference on research and development in information retrieval, pages 475–484. find
  3. Jens Allwood. 2000. An activity-based approach to pragmatics. Gothenburg Papers in Theorecial Linguistics, 76. ↗
  4. Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604. ↗
  5. Luciana Benotti. 2009. Clarification potential of instructions. In Proceedings of the SIGDIAL 2009 Conference, pages 196–205, London, UK. Association for Computational Linguistics. find
  6. Luciana Benotti and Patrick Blackburn. 2017. Modeling the clarification potential of instructions: Predicting clarification requests and other reactions. Computer Speech & Language, 45:536–551. ↗
  7. Luciana Benotti and Patrick Blackburn. 2021. A recipe for annotating grounded clarifications. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4065–4077, Online. Association for Computational Linguistics. ↗
  8. Dan Bohus and Alexander I. Rudnicky. 2005. Sorry and I didn’t catch that! - an investigation of non-understanding errors and recovery strategies. In Proceedings of the 6th SIGdial Workshop on Discourse and Dialogue, pages 128–143, Lisbon, Portugal. Special Interest Group on Discourse and Dialogue (SIGdial). find
  9. Felix Gervits, Antonio Roque, Gordon Briggs, Matthias Scheutz, and Matthew Marge. 2021. How should agents ask questions for situated learning? an annotated dialogue corpus. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 353–359, Singapore and Online. Association for Computational Linguistics. ↗
  10. Pavel Braslavski, Denis Savenkov, Eugene Agichtein, and Alina Dubatovka. 2017. What do you mean exactly? analyzing clarification questions in cqa. In Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval, CHIIR ’17, page 345–348, New York, NY, USA. Association for Computing Machinery. find
  11. Jonathan Ginzburg. 2012. Grounding and CRification. In The Interactive Stance. Oxford University Press, Oxford. ↗
  12. Herbert H Clark. 1996. Using language. Cambridge university press. find
  13. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. ↗
  14. Marco De Boni and Suresh Manandhar. 2003. An analysis of clarification dialogue for question answering. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pages 48–55. find
  15. Robin Deits, Stefanie Tellex, Pratiksha Thaker, Dimitar Simeonov, Thomas Kollar, and Nicholas Roy. 2013. Clarifying commands with information-theoretic human-robot dialog. J. Hum.-Robot Interact., 2(2):58–79. ↗
  16. Alaaeldin El-Nouby, Shikhar Sharma, Hannes Schulz, Devon Hjelm, Layla El Asri, Samira Ebrahimi Kahou, Yoshua Bengio, and Graham W.Taylor. 2019. Tell, draw, and repeat: Generating and modifying images based on continual linguistic instruction. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 10303–10311. find
  17. Tsu-Jui Fu, Xin Wang, Scott T. Grafton, Miguel P. Eckstein, and William Yang Wang. 2020. Iterative language-based image editing via self-supervised counterfactual reasoning. In EMNLP. find
  18. Malte Gabsdil. 2003. Clarification in spoken dialogue systems. In Proceedings of the 2003 AAAI Spring Symposium. Workshop on Natural Language Generation in Spoken and Written Dialogue, pages 28–35. find
  19. Xiaofeng Gao, Qiaozi Gao, Ran Gong, Kaixiang Lin, Govind Thattai, and Gaurav S Sukhatme. 2022. Dialfred: Dialogue-enabled agents for embodied instruction following. IEEE Robotics and Automation Letters, 7(4):10049–10056. ↗
  20. Spandana Gella, Aishwarya Padmakumar, Patrick Lange, and Dilek Hakkani-Tur. 2022. Dialog acts for task driven embodied agents. In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 111–123, Edinburgh, UK. Association for Computational Linguistics.
  21. Jonathan Ginzburg and Andy Luecking. 2021. Requesting clarifications with speech and gestures. In Proceedings of the 1st Workshop on Multimodal Semantic Representations (MMSR), pages 21–31, Groningen, Netherlands (Online). Association for Computational Linguistics. find
  22. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778. ↗
  23. Patrick GT Healey, Matthew Purver, James King, Jonathan Ginzburg, and Greg J Mills. 2003. Experimenting with clarification in dialogue. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 25. find
  24. Xiang Hu, Zujie Wen, Yafang Wang, Xiaolong Li, and Gerard de Melo. 2020. Interactive question clarification in dialogue via reinforcement learning. In Proceedings of the 28th International Conference on Computational Linguistics: Industry Track, pages 78–89, Online. International Committee on Computational Linguistics. find
  25. Ryan Blake Jackson and Tom Williams. 2018. Robot: Asker of questions and changer of norms. Proceedings of ICRES. find
  26. Ryan Blake Jackson and Tom Williams. 2019. Language-capable robots may inadvertently weaken human moral norms. In 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 401–410. IEEE. find
  27. Jin-Hwa Kim, Nikita Kitaev, Xinlei Chen, Marcus Rohrbach, Byoung-Tak Zhang, Yuandong Tian, Dhruv Batra, and Devi Parikh. 2019. CoDraw: Collaborative drawing as a testbed for grounded goal-driven communication. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6495–6513, Florence, Italy. Association for Computational Linguistics.
  28. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. ↗
  29. Julia Kiseleva, Ziming Li, Mohammad Aliannejadi, Shrestha Mohanty, Maartje ter Hoeve, Mikhail Burtsev, Alexey Skrynnik, Artem Zholus, Aleksandr Panov, Kavya Srinet, et al. 2021. Neurips 2021 competition iglu: Interactive grounded language understanding in a collaborative environment. arXiv preprint arXiv:2110.06536. ↗
  30. Koh Mitsuda, Ryuichiro Higashinaka, Yuhei Oga, and Sen Yoshida. 2022. Dialogue collection for recording the process of building common ground in a collaborative task. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 5749–5758, Marseille, France. European Language Resources Association. find
  31. Julia Kiseleva, Alexey Skrynnik, Artem Zholus, Shrestha Mohanty, Negar Arabzadeh, Marc-Alexandre Côté, Mohammad Aliannejadi, Milagro Teruel, Ziming Li, Mikhail Burtsev, et al. 2022. Iglu 2022: Interactive grounded language understanding in a collaborative environment at neurips 2022. arXiv preprint arXiv:2205.13771. ↗
  32. Shrestha Mohanty, Negar Arabzadeh, Milagro Teruel, Yuxuan Sun, Artem Zholus, Alexey Skrynnik, Mikhail Burtsev, Kavya Srinet, Aleksandr Panov, Arthur Szlam, Marc-Alexandre Côté, and Julia Kiseleva. 2022. Collecting interactive multi-modal datasets for grounded language understanding. ↗
  33. Theodora Koulouri and Stanislao Lauria. 2009. Exploring miscommunication and collaborative behaviour in human-robot interaction. In Proceedings of the SIGDIAL 2009 Conference, pages 111–119, London, UK. Association for Computational Linguistics. find
  34. Anjali Narayan-Chen, Prashant Jayannavar, and Julia Hockenmaier. 2019. Collaborative dialogue in Minecraft. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5405–5415, Florence, Italy. Association for Computational Linguistics.
  35. Vaibhav Kumar and Alan W Black. 2020. ClarQ: A large-scale and diverse dataset for clarification question generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7296–7301, Online. Association for Computational Linguistics. find
  36. Charlotte Lambert, Ariel Cordes, Elli Kaplan, Prashant Jayannavar, and Julia Hockenmaier. 2019. Virtual world context encoding for grounded dialogue in minecraft. find
  37. Hyunhee Lee, Gyeongmin Kim, Yuna Hur, and Heuiseok Lim. 2021. Visual thinking of neural networks: Interactive text to image synthesis. IEEE Access, 9:64510–64523. ↗
  38. Zhenhuan Liu, Jincan Deng, Liang Li, Shaofei Cai, Qianqian Xu, Shuhui Wang, and Qingming Huang. 2020. Ir-gan: Image manipulation with linguistic instruction by increment reasoning. Proceedings of the 28th ACM International Conference on Multimedia. find
  39. Bodhisattwa Prasad Majumder, Sudha Rao, Michel Galley, and Julian McAuley. 2021. Ask what’s missing and what’s useful: Improving clarification question generation using global knowledge. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4300–4312, Online. Association for Computational Linguistics. find
  40. Matthew Marge and Alexander Rudnicky. 2015. Miscommunication recovery in physically situated dialogue. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 22–31, Prague, Czech Republic. Association for Computational Linguistics. find
  41. Shoya Matsumori, Yukikoko Abe, Kosuke Shingyouchi, Komei Sugiura, and Michita Imai. 2021. Lattegan: Visually guided language attention for multi-turn text-conditioned image manipulation. IEEE Access, 9:160521–160532. ↗
  42. Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan-Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur. 2022. Teach: Task-driven embodied agents that chat. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2017–2025. find
  43. Matthew Purver, Jonathan Ginzburg, and Patrick Healey. 2003. On the means for clarification in dialogue. In Current and new directions in discourse and dialogue, pages 235–255. Springer. ↗
  44. Sudha Rao and Hal Daumé III. 2018. Learning to ask good questions: Ranking clarification questions using neural expected value of perfect information. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2737–2746, Melbourne, Australia. Association for Computational Linguistics. find
  45. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  46. Verena Rieser, Ivana Kruijff-Korbayová, and Oliver Lemon. 2005. A corpus collection and annotation framework for learning multimodal clarification strategies. In Proceedings of the 6th SIGdial Workshop on Discourse and Dialogue, pages 97–106, Lisbon, Portugal. Special Interest Group on Discourse and Dialogue (SIGdial). find
  47. Verena Rieser and Oliver Lemon. 2006. Using machine learning to explore human multimodal clarification strategies. In Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions, pages 659–666, Sydney, Australia. Association for Computational Linguistics. find
  48. Verena Rieser and Johanna Moore. 2005. Implications for generating clarification requests in task-oriented dialogues. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 239–246, Ann Arbor, Michigan. Association for Computational Linguistics.
  49. Kepa Joseba Rodríguez and David Schlangen. 2004. Form, intonation and function of clarification requests in german task-oriented spoken dialogues. In Proceedings of Catalog (the 8th workshop on the semantics and pragmatics of dialogue; SemDial04). find
  50. Michael Roth, Talita Anthonio, and Anna Sauer. 2022. SemEval-2022 task 7: Identifying plausible clarifications of implicit and underspecified phrases in instructional texts. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1039–1049, Seattle, United States. Association for Computational Linguistics. find
  51. Takaya Saito and Marc Rehmsmeier. 2015. The precision-recall plot is more informative than the roc plot when evaluating binary classifiers on imbalanced datasets. PloS one, 10(3):e0118432. ↗
  52. David Schlangen. 2004. Causes and strategies for requesting clarification in dialogue. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue at HLT-NAACL 2004, pages 136–143, Cambridge, Massachusetts, USA. Association for Computational Linguistics. find
  53. David Schlangen and Raquel Fernández. 2007a. Beyond repair – testing the limits of the conversational repair system. In Proceedings of the 8th SIGdial Workshop on Discourse and Dialogue, pages 51–54, Antwerp, Belgium. Association for Computational Linguistics. find
  54. Svetlana Stoyanchev, Alex Liu, and Julia Hirschberg. 2013. Modelling human clarification strategies. In Proceedings of the SIGDIAL 2013 Conference, pages 137–141, Metz, France. Association for Computational Linguistics. find
  55. Svetlana Stoyanchev, Alex Liu, and Julia Hirschberg. 2014. Towards natural clarification questions in dialogue systems. In AISB symposium on questions, discourse and dialogue, volume 20. find
  56. Svetlana Stoyanchev, Alex Liu, and Julia Bell Hirschberg. 2012. Clarification questions with feedback. ↗
  57. Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. 2020. Vision-and-dialog navigation. In Conference on Robot Learning, pages 394–406. PMLR. find
  58. Tianhao Zhang, Hung-Yu Tseng, Lu Jiang, Weilong Yang, Honglak Lee, and Irfan Essa. 2021. Text as neural operator: Image manipulation by text instruction. Proceedings of the 29th ACM International Conference on Multimedia. find
  59. C Lawrence Zitnick and Devi Parikh. 2013. Bringing semantics into focus using visual abstraction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3009–3016. find
  60. C Lawrence Zitnick, Devi Parikh, and Lucy Vanderwende. 2013. Learning the visual interpretation of sentences. In Proceedings of the IEEE International Conference on Computer Vision, pages 1681–1688. find
  61. David Schlangen and Raquel Fernández. 2007b. Speaking through a noisy channel-experiments on inducing clarification behaviour in human-human dialogue. Proceedings of Interspeech 2007. find
  62. Julian Schlöder and Raquel Fernández. 2014. Clarification requests at the level of uptake. In Proceedings of the 18th Workshop on the Semantics and Pragmatics of Dialogue - Poster Abstracts, Edinburgh, United Kingdom. SEMDIAL. find
  63. Julian J. Schlöder and Raquel Fernández. 2015. Clarifying intentions in dialogue: A corpus study. In Proceedings of the 11th International Conference on Computational Semantics, pages 46–51, London, UK. Association for Computational Linguistics. find
  64. Zhengxiang Shi, Yue Feng, and Aldo Lipani. 2022. Learning to execute actions or ask clarification questions. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2060–2070, Seattle, United States. Association for Computational Linguistics. *Figure 6: Instructions for the first annotator.* find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/madureira-2023-instruction-clarification-requests-multimodal-collaborative/