Elamur Research⌕
☰
home / papers / 2022 / Draw Me a Flower: Processing and Grounding Abstraction in Natural Language

Draw Me a Flower: Processing and Grounding Abstraction in Natural Language

Paper · 2022 · Royi Lachmy, Valentina Pyatkin, Avshalom Manevich et al.

Summary

People giving complex instructions often use abstraction, letting one expression cover many concrete instances through shapes, repetitions or conditions. NLP has lacked benchmarks that deliberately capture this. The authors address a practical obstacle: crowdworkers are unfamiliar with formal abstraction, and teaching them about it would make their language less natural. To get around this, they borrow ideas from Computational Thinking research in STEM education. They design visual stimuli that encourage higher-order descriptions without asking for them directly. These stimuli are built into HEXAGONS, a collaborative 2D drawing game in which an Instructor describes a target pattern that must be reproduced on a hexagonal board.

The resulting corpus contains over 4k grounded instructions covering objects, control structures and functions. From it the authors derive an instruction-to-execution task. They compare a naïve rule-based baseline with classification-based and generation-based neural models. The neural systems beat the baseline but remain far below human performance. Their accuracy also falls as the level of abstraction rises, regardless of model, setup or amount of training data. This establishes abstraction as a measurable open problem for grounded language understanding.

Key points

  • HEXAGONS dataset: more than 4k naturally elicited, visually grounded instructions on a 180-tile board, annotated by abstraction level. The data, models, game and collection app are public at OnlpLab.github.io/Hexagons.
  • Models evaluated: a rule-based baseline and neural classification and generation approaches, including DeBERTa. They are scored with F1 and EM under Full-History and No-History input settings.
  • Findings:
    • Performance correlates inversely with abstraction level.
    • Concrete instructions are executed well, while repetitions combined with conditions are handled poorly.
    • Supplying a longer instruction history improves results.
  • Comparison with the 2-D Blocks dataset: in a sample of 594 instructions (5% of its train set), 96.5% simply move a single block. Only 2.7% show low-level abstraction and 0.8% show mid-level abstraction.
  • Funding: the research was supported by ERC-StG grant no. 677352.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Allen Institute for Artificial IntelligenceSeattle · United States275 works · 97,634 citations🎓Bar-Ilan UniversityRamat Gan · Israel141 works · 23,225 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Royi Lachmy, Valentina Pyatkin, Avshalom Manevich and Reut Tsarfaty. 2022. Draw Me a Flower: Processing and Grounding Abstraction in Natural Language. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00522

@article{lachmy2022draw,
  title = {{Draw Me a Flower: Processing and Grounding Abstraction in Natural Language}},
  author = {Royi Lachmy, Valentina Pyatkin, Avshalom Manevich and Reut Tsarfaty},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.1162/tacl\_a\_00522},
  url = {https://doi.org/10.1162/tacl_a_00522}
}

References

48 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anne H. Anderson, Miles Bader, Ellen Gurman Bard, Elizabeth Boyle, Gwyneth Doherty, Simon Garrod, Stephen Isard, Jacqueline Kowtko, Jan McAllister, Jim Miller, Catherine Sotillo, Henry S. Thompson, and Regina Weinert. 1991. The HCRC map task corpus. Language and Speech, 34(4):351–366. https://doi.o… ↗
  2. Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel. 2018. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. 2018 IEEE/CVF Conference on Computer Vision and Pa…
  3. Satabdi Basu, Daisy W. Rutstein, Yuning Xu, Haiwen Wang, and Linda Shear. 2021. A principled approach to designing computational thinking concepts and practices assessments for upper elementary grades. Computer Science Education, 31(2):169–198. https://doi.org/10.1080/08993408.2020.1866939 — [doi:10.1080/08993408.2020.1866939](https://doi.org/10.1080/08993408.2020.1866939) ↗
  4. Yonatan Bisk, Daniel Marcu, and William Wong. 2016a. Towards a dataset for human computer communication via grounded language acquisition. In Symbiotic Cognitive Systems, Papers from the 2016 AAAI Workshop, Phoenix, Arizona, USA, February 13, 2016, volume WS-16-14 of AAAI Workshops. AAAI Press. find
  5. Yonatan Bisk, Kevin J. Shih, Yejin Choi, and Daniel Marcu. 2018. Learning interpretable spatial operations in a rich 3d blocks world. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and… find
  6. Yonatan Bisk, Deniz Yuret, and Daniel Marcu. 2016b. Natural language communication with robots. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 751–761, San Diego, California. Association for Com… find
  7. Erin M. Burgoon, Marlone D. Henderson, and Arthur B. Markman. 2013. There are many ways to see the forest for the trees: A tour guide for abstraction. Perspectives on Psychological Science, 8(5):501–520. https://doi.org/10.1177/1745691613497964, PubMed: 26173209 — [doi:10.1177/1745691613497964](https://doi.org/10.1177/1745691613497964) ↗
  8. Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. 2019. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12530–12539. https://doi.org /10.1109/CVPR.201… ↗
  9. Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. 2018. BabyAI: A platform to study the sample efficiency of grounded language learning. In International Conference on Learning Representations. find
  10. Herbert H. Clark and Deanna Wilkes-Gibbs. 1986. Referring as a collaborative process. Cognition, 22(1):1–39. https://doi.org/10.1016 /0010-0277(86)90010-7 find
  11. Jan Cuny, Larry Snyder, and Jeannette M. Wing. 2010. Demystifying Computational Thinking for Non-computer Scientists. Unpublished manuscript in progress, referenced in http://www.cs.cmu.edu/∼CompThink/resources/TheLinkWing.pdf Peter J. Denning, Douglas E. Comer, David Gries, Michael C. Mulder, All… — [doi:10.1145/63238.63239](https://doi.org/10.1145/63238.63239) ↗
  12. Edsger W. Dijkstra. 1972. The humble programmer. ACM Turing Award Lectures. https://doi.org/10.1145/1283920.1283927 — [doi:10.1145/1283920.1283927](https://doi.org/10.1145/1283920.1283927) ↗
  13. Edsger W. Dijkstra. Accessed 1 May 2021. Ew dijkstra quotes. find
  14. Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. 2018. Improving text-to-SQL evaluation methodology. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages…
  15. Nicholas FitzGerald, Julian Michael, Luheng He, and Luke Zettlemoyer. 2018. Large-scale QA-SRL parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2051–2060, Melbourne, Australia. Association for Computational Linguistics…
  16. David Ginat and Yoav Blau. 2017. Multiple levels of abstraction in algorithmic problem solving. In Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education, SIGCSE ’17, pages 237–242, New York, NY, USA. Association for Computing Machinery. https://doi.org/10 .1145/3017680… find
  17. Omer Goldman, David Guriel, and Reut Tsarfaty. 2022. (un)solving morphological inflection: Lemma overlap artificially inflates models’ performance. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 864–870, Dublin, Ireland. Ass… find
  18. Herbert P. Grice. 1975. Logic and conversation. In Speech Acts, pages 41–58. Brill. https://doi.org/10.1163/9789004368811 003 ↗
  19. Shuchi Grover and Roy Pea. 2013. Computational thinking in k–12: A review of the state of the field. Educational Researcher, 42(1):38–43. https://doi.org/10.3102/0013189X124 63051 — [doi:10.3102/0013189x124](https://doi.org/10.3102/0013189x124) ↗
  20. Kilem L. Gwet. 2015. On Krippendorff’s alpha coefficient. Accessed: 1 June 2022. find
  21. Janosch Haber, Tim Baumgärtner, Ece Takmaz, Lieke Gelderloos, Elia Bruni, and Raquel Fernández. 2019. The PhotoBook dataset: Building common ground through visually-grounded dialogue. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1895–1910, Flore…
  22. Bruria Haberman. 2004. High-school students’ attitudes regarding procedural abstraction. Education and Information Technologies, 9(2):131–145. https://doi.org/10.1023/B:EAIT.0000027926.99053.6f Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with dis… ↗
  23. Jonathan Herzig and Jonathan Berant. 2020. Span-based semantic parsing for compositional generalization. arXiv preprint arXiv:2009.06040. Version 2. find
  24. Prashant Jayannavar, Anjali Narayan-Chen, and Julia Hockenmaier. 2020. Learning to execute instructions in a Minecraft dialogue. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2589–2602, Online. Association for Computational Linguistics. https://doi… find
  25. Fereshte Khani, Noah Goodman, and Percy Liang. 2018. Planning, inference, and pragmatics in sequential language games. Transactions of the Association for Computational Linguistics, 6:543–555. https://doi.org/10.1162/tacl_a_00037 ↗
  26. Jin-Hwa Kim, Nikita Kitaev, Xinlei Chen, Marcus Rohrbach, Byoung-Tak Zhang, Yuandong Tian, Dhruv Batra, and Devi Parikh. 2019. CoDraw: Collaborative drawing as a testbed for grounded goal-driven communication. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics…
  27. Herman Koppelman and Betsy Van Dijk. 2010. Teaching abstraction in introductory courses. In Proceedings of the Fifteenth Annual Conference on Innovation and Technology in Computer Science Education, pages 174–178. https://doi.org/10.1145/1822090 .1822140 — [doi:10.1145/1822090](https://doi.org/10.1145/1822090) ↗
  28. Klaus Krippendorff. 2004. Content Analysis: An Introduction to Its Methodology (2nd ed.), Chapter 11. Beverly Hills, CA: Sage. find
  29. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586. Version 1.
  30. Reginald Long, Panupong Pasupat, and Percy Liang. 2016. Simpler context-dependent logical forms via model projections. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1456–1465, Berlin, Germany. Association for Computational L… find
  31. Matt MacMahon, Brian Stankiewicz, and Benjamin Kuipers. 2006. Walk the talk: Connecting language, knowledge, and action in route instructions. In Proceedings of the 21st National Conference on Artificial Intelligence -Volume 2, AAAI’06, pages 1475–1482. AAAI Press. find
  32. Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi. 2018. Mapping instructions to actions in 3D environments with visual goal prediction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2667–2678, Brussels, Be… find
  33. Tzuf Paz-Argaman and Reut Tsarfaty. 2019. RUN through the streets: A new dataset and baseline models for realistic urban navigation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EM… find
  34. Bedřich Pišl and David Mareček. 2017. Communication with robots using multilayer recurrent networks. In Proceedings of the First Workshop on Language Grounding for Robotics, pages 44–48. https://doi.org/10.18653/v1/W17-2806 ↗
  35. Valentina Pyatkin, Ayal Klein, Reut Tsarfaty, and Ido Dagan. 2020. QADiscourse - discourse relations as QA pairs: Representation, crowdsourcing and baselines. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2804–2819, Online. Association for C…
  36. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.
  37. Emily Relkin and Marina Umaschi Bers. 2019. Designing an assessment of computational thinking abilities for young children. In Lynn E. Cohen and Sandra Waite-Stupiansky, editors, STEM in Early Childhood Education: How Science, Technology, Engineering, and Mathematics Strengthen Learning, chapter 5.… — [doi:10.4324/9780429453755-5](https://doi.org/10.4324/9780429453755-5) find
  38. Paul Roit, Ayal Klein, Daniela Stepanov, Jonathan Mamou, Julian Michael, Gabriel Stanovsky, Luke Zettlemoyer, and Ido Dagan. 2020. Controlled crowdsourcing for high-quality QA-SRL annotation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7008–7013,…
  39. Liliana Ructtinger and Robert Stevens. 2017. Computational thinking task design and assessment. Scan: The Journal For Educators, 36(1):34–41. find
  40. Valerie J. Shute, Chen Sun, and Jodi Asbell-Clarke. 2017. Demystifying computational thinking. Educational Research Review, 22:142–158. https://doi.org/10.1016/j.edurev.2017.09.003 ↗
  41. Alane Suhr, Claudia Yan, Jack Schluger, Stanley Yu, Hadi Khader, Marwa Mouallem, Iris Zhang, and Yoav Artzi. 2019. Executing instructions in situated collaborative interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint…
  42. Takuma Udagawa and Akiko Aizawa. 2019. A natural language corpus of common grounding under continuous and partially-observable context. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):7120–7127. https://doi.org/10.1609/aaai.v33i01 .33017120 ↗
  43. Sida I. Wang, Samuel Ginn, Percy Liang, and Christopher D. Manning. 2017. Naturalizing a programming language via interactive learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 929–938, Vancouver, Canada. Association fo… find
  44. Sida I. Wang, Percy Liang, and Christopher D. Manning. 2016. Learning language games through interaction. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2368–2378, Berlin, Germany. Association for Computational Linguistics. h… find
  45. Jeannette Wing. 2017. Computational thinking’s influence on research and education for all. Italian Journal of Educational Technology, 25(2):7–14. find
  46. Jeannette M. Wing. 2011. Computational thinking—whatandwhy? CMUResearchNotebook. find
  47. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… — Transformers: State-of-the-Art Natural Language Processing
  48. Jing Xu, Arthur Szlam, and Jason Weston. 2022. Beyond goldfish memory: Long-term open-domain conversation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5180–5197, Dublin, Ireland. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lachmy-2022-draw-me-flower-processing-grounding/