Elamur Research⌕
☰
home / papers / 2026 / To Skip, to Swap or to Not Swap? Identifying Step Transition Types in Instructional Manuals

To Skip, to Swap or to Not Swap? Identifying Step Transition Types in Instructional Manuals

Summary

LLMs are increasingly used to guide people through everyday procedures. In assistive settings, though, a good plan must bend: some steps can be dropped, some reordered, and others must keep a fixed order. This work tests whether models can recognise such flexible relations in written instructions. The authors propose SIO (Sequential-Interchangeable-Optional), an evaluation framework that treats each relation as a local binary decision. Does an adjacent pair need to keep its order? Can the two steps after an anchor step be swapped? Can the step right after the anchor be skipped? Human annotators labelled WikiHow task-parts from WikiHowClean in five physical-activity domains. Eleven LLMs from seven open and proprietary families were compared against a heuristic baseline built on a visually-aware probabilistic procedural knowledge graph (PKG). Larger models trained on more data identify step types better, but the task stays hard, especially for interchangeable and optional steps. LLMs also give contradictory sequential/interchangeable judgements and break output constraints, while the symbolic PKG is more consistent. The resource helps assess LLMs as reliable, adaptive procedural assistants.

Key points

  • SIO dataset: 298 WikiHowClean task-parts (15 dev, 283 test). The five domains covered are Home and Garden, Cars and Other Vehicles, Food and Entertaining, Computers and Electronics, and Hobbies and Crafts. The Cars and Computers domains are absent from the dev set. The dataset is released on GitHub (Mallory24/SIO_dataset).
  • Annotation: 21 student annotators were paid €13/hour, and each task was labelled by three of them. Expert-knowledge instances are rare: 124 for sequential, 1 for interchangeable and 3 for optional. Commonsense instances number 778, 129 and 82 respectively.
  • Models evaluated: the LLM families include Llama-2 (7B/13B/70B), Llama-3.1 (8B/70B), Gemma-2 (2B/9B), Mistral-7B and Mixtral-8x7B. They are compared with the PKG-based heuristic system.
  • Instruction compliance is weak for small models. Gemma-2-2B violates task constraints in 81.63% of interchangeable cases and 82.33% of optional cases. Llama-2-7B does so in 74.91% of interchangeable cases.
  • Rationales: the authors prompted models to justify their answers through action pre- and postconditions. The strongest LLMs produced explanations that suggest some awareness of entity states.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/5jvvjyarv88s

@inproceedings{yang2026skip,
  title = {{To Skip, to Swap or to Not Swap? Identifying Step Transition Types in Instructional Manuals}},
  author = {Yang, Hsiu-Yu and Roth, Michael and Bulling, Andreas and Silberer, Carina},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/5jvvjyarv88s},
  url = {https://doi.org/10.63317/5jvvjyarv88s}
}

References

36 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Talita Anthonio and Michael Roth. 2021. Resolving implicit references in instructional texts. pages 58–71, Punta Cana, Dominican Republic and Online. find
  2. Kumar Ashutosh, Santhosh Kumar Ramakrishnan, Triantafyllos Afouras, and Kristen Grauman. 2023. Video-mined task graphs for keystep recognition in instructional videos. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 20… find
  3. Yuwei Bao, Keunwoo Yu, Yichi Zhang, Shane Storks, Itamar Bar-Yossef, Alex de la Iglesia, Megan Su, Xiao Zheng, and Joyce Chai. 2023. Can foundation models watch, talk and guide you step by step to make a cake? pages 12325– 12341, Singapore. find
  4. Faeze Brahman, Chandra Bhagavatula, Valentina Pyatkin, Jena D Hwang, Xiang Lorraine Li, Hirona J Arai, Soumya Sanyal, Keisuke Sakaguchi, Xiang Ren, and Yejin Choi. 2023. PlaSma: Making small language models better procedural knowledge models for (counterfactual) planning. arXiv [cs.CL]. find
  5. Lin Guan, Karthik Valmeekam, Sarath Sreedharan, and Subbarao Kambhampati. 2023. Leveraging pre-trained large language models to construct and utilize world models for model-based task planning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processin… find
  6. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. find
  7. Yunseok Jang, Sungryull Sohn, Lajanugen Logeswaran, Tiange Luo, Moontae Lee, and Honglak Lee. 2023. Multimodal subtask graph generation from instructional videos. arXiv [cs.LG]. find
  8. Najoung Kim and Sebastian Schuster. 2023. Entity tracking in language models. pages 3835–3855, Toronto, Canada. find
  9. Mahnaz Koupaee and William Yang Wang. 2018. WikiHow: A large scale text summarization dataset. arXiv [cs.CL]. find
  10. Yash Kumar Lal, Vanya Cohen, Nathanael Chambers, Niranjan Balasubramanian, and Ray Mooney. 2024. CaT-bench: Benchmarking language model understanding of causal and temporal dependencies in plans. pages 19336–19354, Miami, Florida, USA. find
  11. Bill Yuchen Lin, Chengsong Huang, Qian Liu, Wenda Gu, Sam Sommerer, and Xiang Ren. 2023. On grounded planning for embodied tasks with language models. In Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligenc… find
  12. Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. 2019a. End-to-end learning of visual representations from uncurated instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9879–9889. find
  13. Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. 2019b. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2630–2640. find
  14. Dhruvesh Patel, Hamid Eghbalzadeh, Nitin Kamra, Michael Louis Iuzzolino, Unnat Jain, and Ruta Desai. 2023. Pretrained language models as visual planners for human assistance. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023. find
  15. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,… find
  16. Kyeongmin Rim, Jingxuan Tu, Bingyang Ye, Marc Verhagen, Eben Holderness, and James Pustejovsky. 2023. The coreference under transformation labeling dataset: Entity tracking in procedural texts using event models. pages 12448– 12460, Toronto, Canada. find
  17. Keisuke Sakaguchi, Chandra Bhagavatula, Ronan Le Bras, Niket Tandon, Peter Clark, and Yejin Choi. 2021. proscript: Partially ordered scripts generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2138–2149. find
  18. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv [cs.LG]. find
  19. Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. 2022. LLM-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 2998-3009, pages 2998–3009… find our card
  20. Niket Tandon, Keisuke Sakaguchi, Bhavana Dalvi, Dheeraj Rajagopal, Peter Clark, Michal Guerquin, Kyle Richardson, and Eduard Hovy. 2020. A dataset for tracking entities in open domain procedural text. pages 6408–6417, Online. find
  21. Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2023a. PlanBench: An extensible benchmark for evaluating large language models on planning and reasoning about change. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Ben… find our card
  22. Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. 2023b. On the planning abilities of large language models : A critical investigation. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 202… find
  23. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… find our card
  24. Te-Lin Wu, Alex Spangher, Pegah Alipoormolabashi, Marjorie Freedman, Ralph Weischedel, and Nanyun Peng. 2022. Understanding multimodal procedural knowledge by sequencing multimodal instructional manuals. pages 4525–4542, Dublin, Ireland. find
  25. Te-Lin Wu, Caiqi Zhang, Qingyuan Hu, Alexander Spangher, and Nanyun Peng. 2023a. Learning action conditions from instructional manuals for instruction understanding. pages 3023–3043, Toronto, Canada. find
  26. Xueqing Wu, Sha Li, and Heng Ji. 2023b. OpenPI-C: A better benchmark and stronger baseline for open-vocabulary state tracking. pages 7213– 7222, Toronto, Canada. find
  27. Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. 2021. VideoCLIP: Contrastive pre-training for zero-shot video-text understanding. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processin… find
  28. Hsiu-Yu Yang and Carina Silberer. 2023. Implicit affordance acquisition via causal Action–Effect modeling in the video domain. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computatio… find
  29. Yue Yang, Artemis Panagopoulou, Qing Lyu, Li Zhang, Mark Yatskar, and Chris Callison-Burch. 2021. Visual goal-step inference using wikiHow. pages 2167–2179, Online and Punta Cana, Dominican Republic. find
  30. Huibin Zhang, Zhengkun Zhang, Yao Zhang, Jun Wang, Yufan Li, Ning Jiang, Xin Wei, and Zhenglu Yang. 2022. Modeling temporal-modal entity graph for procedural multimodal machine comprehension. find
  31. Li Zhang. 2022. Reasoning about procedures with natural language processing: A tutorial. arXiv [cs.CL]. find
  32. Li Zhang, Qing Lyu, and Chris Callison-Burch. 2020. Reasoning about goals, steps, and temporal ordering with WikiHow. pages 4630–4639, Online. find
  33. Li Zhang, Hainiu Xu, Abhinav Kommula, Chris Callison-Burch, and Niket Tandon. 2024. OpenPI2.0: An improved dataset for entity tracking in texts. pages 166–178, St. Julian’s, Malta. find
  34. Honglu Zhou, Roberto Martín-Martín, Mubbasir Kapadia, Silvio Savarese, and Juan Carlos Niebles. 2023a. Procedure-aware pretraining for instructional video understanding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023. find
  35. Shuyan Zhou, Li Zhang, Yue Yang, Qing Lyu, Pengcheng Yin, Chris Callison-Burch, and Graham Neubig. 2022. Show me more details: Discovering hierarchies of procedures from semi-structured web data. pages 2998–3012, Dublin, Ireland. find
  36. Yu Zhou, Sha Li, Manling Li, Xudong Lin, Shih-Fu Chang, Mohit Bansal, and Heng Ji. 2023b. Non-sequential graph script induction via multimedia grounding. pages 5529–5545, Toronto, Canada. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2026-skip-swap-or-not-swap/