Elamur Research⌕
☰
home / papers / 2020 / Mapping Natural Language Instructions to Mobile UI Action Sequences

Mapping Natural Language Instructions to Mobile UI Action Sequences

Paper · 2020 · Yang Li, Jiacong He, Xin Zhou et al.

Summary

The paper introduces a new grounding task: converting a multi-step English instruction, such as a help-page recipe for changing Wi-Fi settings, into actions that can be executed directly on an Android touchscreen. Instruction–action pairs are hard to collect at scale, so the authors split the problem into two stages, each trained on its own data. First, a Transformer reads the instruction and extracts a tuple for each step, made up of operation, object and argument phrases. Second, a grounding Transformer encodes every UI object on the current screen, using both its content and its position, and links the object phrase to the right element. Training data comes from two new resources: AndroidHowTo, web how-to texts annotated with action spans, and RicoSCA, synthetic commands built over the Rico screen corpus. A third set, PixelHelp, holds human-executed tasks and is used only for end-to-end testing. On PixelHelp, the combined system fully reproduces the correct action sequence in 70.59% of tasks. This makes the work a foundation for voice- or text-driven phone automation, which is especially relevant for accessibility.

Key points

  • PixelHelp: 187 multi-step instructions taken from Pixel Phone Help pages, covering general, Gmail, Chrome and Photos tasks. Annotators carried out each task on an emulator, and the recorded tasks range from 2 to 8 steps.
  • AndroidHowTo: 32,436 annotated data points drawn from 9,893 unique web instructions. Each instruction was labeled by three annotators, and all test items have full agreement among them. A Transformer using sum-pooled span representations matched the complete span sequence in 85.56% of cases.
  • RicoSCA: about 295K synthetic single-step commands covering 178K UI objects on 25K screens. It is derived from Rico, which contains 72K screens from 9.7K apps.
  • End-to-end results on PixelHelp: 89.21% partial accuracy and 70.59% complete accuracy. Phrase extraction failed on 14 tasks, mostly by adding extra steps (11 tasks).
  • Grounding from the decoder's hidden state performed very well on synthetic validation data but transferred poorly to PixelHelp. The authors attribute this to a style gap between synthetic commands and natural instructions. The data pipeline is released as seq2act in the google-research repository.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States353 works · 364,806 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

Cite

Cite as Yang Li, Jiacong He, Xin Zhou, Yuan Zhang and Jason Baldridge. 2020. Mapping Natural Language Instructions to Mobile UI Action Sequences. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.729

@inproceedings{li2020mapping,
  title = {{Mapping Natural Language Instructions to Mobile UI Action Sequences}},
  author = {Yang Li, Jiacong He, Xin Zhou, Yuan Zhang and Jason Baldridge},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.729},
  url = {https://doi.org/10.18653/v1/2020.acl-main.729}
}

References

28 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. 2018. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE Conference on Computer Vis…
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473.
  3. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735– 1780. find
  4. Tianwen Jiang, Tong Zhao, Bing Qin, Ting Liu, Nitesh Chawla, and Meng Jiang. 2019. Multi-input multi-output sequence labeling for joint extraction of fact and condition tuples from scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th… find
  5. S. R. K. Branavan, Harr Chen, Luke S. Zettlemoyer, and Regina Barzilay. 2009. Reinforcement learning for mapping instructions to actions. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNL… find
  6. Huda Khayrallah, Sean Trott, and Jerome Feldman. 2015. Natural language for human robot interaction. find
  7. S.R.K. Branavan, Luke Zettlemoyer, and Regina Barzilay. 2010. Reading between the lines: Learning to map high-level instructions to commands. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1268– 1277, Uppsala, Sweden. Association for Computational L… find
  8. Kenton Lee, Tom Kwiatkowski, Ankur P. Parikh, and Dipanjan Das. 2016. Learning recurrent span representations for extractive question answering. CoRR, abs/1611.01436. — [arXiv:1611.01436](https://arxiv.org/abs/1611.01436) find
  9. David L. Chen and Raymond J. Mooney. 2011. Learning to interpret natural language navigation instructions from observations. In Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence, AAAI’11, pages 859–865. AAAI Press. find
  10. Howard Chen, Alane Suhr, Dipendra Misra, and Yoav Artzi. 2019. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In Conference on Computer Vision and Pattern Recognition. find
  11. Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hi-bschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017. Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th Annual Symposium on User Interface Software and Technology, UIST ’17. find
  12. Izzeddin Gur, Ulrich Rueckert, Aleksandra Faust, and Dilek Hakkani-Tur. 2019. Learning to navigate the web. In International Conference on Learning Representations. find
  13. Luheng He, Kenton Lee, Omer Levy, and Luke Zettlemoyer. 2018. Jointly predicting predicates and arguments in neural semantic role labeling. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 364–369, Melbourne, Australia. Associ…
  14. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188–197, Copenhagen, Denmark. Association for Computational Linguistics.
  15. Yang Li, Lukasz Kaiser, Samy Bengio, and Si Si. 2019. Area attention. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3846–3855, Long Beach, California, USA. PMLR. find
  16. E. Z. Liu, K. Guu, P. Pasupat, T. Shi, and P. Liang. 2018. Reinforcement learning on web interfaces using workflow-guided exploration. In International Conference on Learning Representations (ICLR). find
  17. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421, Lisbon, Portugal. Association for Computational Linguistics.
  18. Ramesh Manuvinakurike, Jacqueline Brixey, Trung Bui, Walter Chang, Doo Soon Kim, Ron Artstein, and Kallirroi Georgila. 2018. Edit me: A corpus and a framework for understanding natural language image editing. In Proceedings of the Eleventh International Conference on Language Resources and Evaluatio… find
  19. Hongyuan Mei, Mohit Bansal, and Matthew R. Walter. 2016. Listen, attend, and walk: Neural mapping of navigational instructions to action sequences. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI’16, pages 2772–2778. AAAI Press. find
  20. Dipendra Misra, John Langford, and Yoav Artzi. 2017. Mapping instructions and visual observations to actions with reinforcement learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1004–1015. find
  21. Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. 2016. Learning convolutional neural networks for graphs. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 2014–2023, New York, New York, USA. PMLR. find
  22. Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. 2018. Image transformer. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4055–4064, Stock-holmsmässan, Stoc… find
  23. Lance Ramshaw and Mitch Marcus. 1995. Text chunking using transformation-based learning. In Third Workshop on Very Large Corpora.
  24. Zhanna Sarsenbayeva. 2018. Situational impairments during mobile interaction. In Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, pages 498–503. find
  25. Alane Suhr, Srinivasan Iyer, and Yoav Artzi. 2018. Learning to map context-dependent sentences to executable formal queries. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), page… find
  26. Richard Szeliski. 2010. Computer Vision: Algorithms and Applications, 1st edition. Springer-Verlag, Berlin, Heidelberg. find
  27. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR, abs/1706.03762.
  28. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2692–2700. Curran Associates, Inc. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2020-mapping-natural-language-instructions-mobile/