PIQA: Reasoning about Physical Commonsense in Natural Language
Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1911.11641v1
Summary
The paper asks whether language models trained only on text can handle everyday physical reasoning, such as choosing the right household object for a task. Such knowledge is rarely written down explicitly because of reporting bias. The authors frame this as a new task and release the PIQA benchmark (Physical Interaction: Question Answering). Each item pairs a goal with two candidate solutions, and exactly one of them is correct. Crowdworkers wrote the items using instructables.com guides as inspiration, which pushed them toward unusual uses of common objects. For each goal they also wrote a subtly broken "trick" solution. Low-agreement items were removed during validation, and the AFLite algorithm filtered out superficial artifacts. Humans solve the benchmark almost perfectly, while strong pretrained models such as RoBERTa lag far behind. Error analysis shows models confuse fine spatial relations and novel object combinations. The work matters because it exposes the limits of text-only learning about the world. It also offers a target for grounded language research relevant to robotics.
Key points
- Human accuracy on PIQA is about 95%, while the best large pretrained models reach roughly 77%.
- The data splits are just over 16,000 training pairs, about 2K development pairs and about 3K test pairs.
- Goals average 7.8 tokens and solutions average 21.3 tokens, giving over 3.7 million training tokens.
- Correct and trick solutions have nearly identical length distributions and share at least 85% of their vocabulary in each part of speech. The training data contains 6,881 unique nouns, 2,493 verbs, 2,263 adjectives and 604 adverbs.
- The prompts were drawn from six instructables sections, namely food, home, workshop, craft, costume and outside. Annotators had to pass a qualification test with over 80% accuracy and were paid more than $15/hr on average.
Cited by
- Fanar: An Arabic-Centric Multimodal Generative AI Platform
- Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
- Yi: Open Foundation Models by 01.AI
- SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
- The Falcon Series of Open Language Models
- PaLM 2 Technical Report
- QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
- SliceGPT: Compress Large Language Models by Deleting Rows and Columns
- Qwen Technical Report
- A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity
- Efficient Training of Language Models to Fill in the Middle
- Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions
- Experience Grounds Language
- GPT-NeoX-20B: An Open-Source Autoregressive Language Model
- Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
- InternLM2 Technical Report
- AlpaGasus: Training A Better Alpaca with Fewer Data
- Moshi: a speech-text foundation model for real-time dialogue
- Investigating Data Contamination in Modern Benchmarks for Large Language Models
- GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
- LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding
- HeSum: a Novel Dataset for Abstractive Text Summarization in Hebrew
- Evaluating Large Language Models: A Comprehensive Survey
- Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey
- Aligning AI With Shared Human Values
- Measuring Massive Multitask Language Understanding
- LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models
- Language Is Not All You Need: Aligning Perception with Language Models
- Lawyer LLaMA Technical Report
- COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs
- Knowledge Unlearning for Mitigating Privacy Risks in Language Models
- UNIFIEDQA: Crossing Format Boundaries with a Single QA System
- Self-Alignment with Instruction Backtranslation
- Textbooks Are All You Need II: phi-1.5 technical report
- DataComp-LM: In search of the next generation of training sets for language models
- Few-shot Learning with Multilingual Language Models
- RA-DIT: Retrieval-Augmented Dual Instruction Tuning
- HeSum: a Novel Dataset for Abstractive Text Summarization in Hebrew
- LLM-QAT: Data-Free Quantization Aware Training for Large Language Models
- LLM-Pruner: On the Structural Pruning of Large Language Models
- ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
- Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs
- Large Language Models: A Survey
- A Comprehensive Overview of Large Language Models
- Large Language Diffusion Models
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
- How Context Affects Language Models' Factual Predictions
- XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning
- Reasoning with Language Model Prompting: A Survey
- Learning How to Ask: Querying LMs with Mixtures of Soft Prompts
- DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale
- SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks
- Leveraging Large Language Models for Multiple Choice Question Answering
- Multitask Prompted Training Enables Zero-Shot Task Generalization
- Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models
- OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models
- Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
- To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning
- Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
- UL2: Unifying Language Learning Paradigms
- Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model
- SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer
- RedPajama: an Open Dataset for Training Large Language Models
- BloombergGPT: A Large Language Model for Finance
- Effective Long-Context Scaling of Foundation Models
- GLM-130B: An Open Bilingual Pre-trained Model
- TinyLlama: An Open-Source Small Language Model
- A Survey of Large Language Models
- Representation Engineering: A Top-Down Approach to AI Transparency
Cite
@misc{bisk2019piqa,
title = {{PIQA: Reasoning about Physical Commonsense in Natural Language}},
author = {Bisk, Yonatan and Zellers, Rowan and Bras, Ronan Le and Gao, Jianfeng and Choi, Yejin},
year = {2019},
doi = {10.48550/arxiv.1911.11641},
eprint = {1911.11641},
archivePrefix = {arXiv},
url = {https://doi.org/10.48550/arxiv.1911.11641}
}References
- Agrawal, P.; Nair, A.; Abbeel, P.; Malik, J.; and Levine, S. 2016. Learning to poke by poking: Experiential learning of intuitive physics. In NeurIPS. find
- Bisk, Y.; Buys, J.; Pichotta, K.; and Choi, Y. 2019. Benchmarking hierarchical script knowledge. In NAACL-HLT. find
- Bosselut, A.; Rashkin, H.; Sap, M.; Malaviya, C.; Celikyilmaz, A.; and Choi, Y. 2019. COMET: Commonsense Transformers for Automatic Knowledge Graph Construction. In ACL. find our card
- Brysbaert, M.; Warriner, A. B.; and Kuperman, V. 2014. Concreteness ratings for 40 thousand generally known english word lemmas. Behavior Research Methods (46):904– 911. find
- Byravan, A.; Leeb, F.; Meier, F.; and Fox, D. 2018. Se3-pose-nets: Structured deep dynamics models for visuomotor planning and control. In ICRA. find
- Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT. find
- Elazar, Y.; Mahabal, A.; Ramachandran, D.; Bedrax-Weiss, T.; and Roth, D. 2019. How large are lions? inducing distributions over quantitative attributes. In ACL. find
- Forbes, M., and Choi, Y. 2017. Verb physics: Relative physical knowledge of actions and objects. In ACL. find
- Gao, Q.; Doering, M.; Yang, S.; and Chai, J. 2016. Physical causality of action verbs in grounded language understanding. In ACL, 1814–1824. find
- Goldberg, Y. 2019. Assessing BERT’s Syntactic Abilities. arXiv:1901.05287. find our card
- Gururangan, S.; Swayamdipta, S.; Levy, O.; Schwartz, R.; Bowman, S.; and Smith, N. A. 2018. Annotation artifacts in natural language inference data. In NAACL-HLT, 107–112. find
- Hespos, S. J., and Spelke, E. S. 2004. Conceptual precursors to language. Nature 430:453–456. find
- Hessel, J.; Mimno, D.; and Lee, L. 2018. Quantifying the visual concreteness of words and topics in multimodal datasets. In NAACL-HLT, 2194–2205. find
- Krishna, R.; Zhu, Y.; Groth, O.; Johnson, J.; Hata, K.; Kravitz, J.; Chen, S.; Kalantidis, Y.; Li, L.-J.; Shamma, D. A.; Bernstein, M.; and Fei-Fei, L. 2016. Visual genome: Connecting language and vision using crowdsourced dense image annotations. In arXiv:1602.07332. find our card
- Li, Y.-L.; Xu, L.; Huang, X.; Liu, X.; Ma, Z.; Chen, M.; Wang, S.; Fang, H.-S.; and Lu, C. 2019. Hake: Human activity knowledge engine. arXiv preprint arXiv:1904.06539. — [arXiv:1904.06539](https://arxiv.org/abs/1904.06539) find
- Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692. find
- Matuszek, C. 2018. Grounded Language Learning: Where Robotics and NLP Meet. In IJCAI, 5687 – 5691. find
- Mottaghi, R.; Rastegari, M.; Gupta, A.; and Farhadi, A. 2016. “what happens if...” learning to predict the effect of forces in images. In Leibe, B.; Matas, J.; Sebe, N.; and Welling, M., eds., ECCV, 269–285. find
- Nair, L.; Balloch, J.; and Chernova, S. 2019. find
- Tool Mac-gyvering: Tool Construction Using Geometric Reasoning. In ICRA. Petroni, F.; Rocktschel, T.; Lewis, P.; Bakhtin, A.; Wu, Y.; Miller, A. H.; and Riedel, S. 2019. Language models as knowledge bases? In EMNLP. find our card
- Poliak, A.; Naradowsky, J.; Haldar, A.; Rudinger, R.; and Van Durme, B. 2018. Hypothesis Only Baselines in Natural Language Inference. In Joint Conference on Lexical and Computational Semantics (StarSem). find
- Radford, A.; Narasimhan, K.; Salimans, T.; and Sutskever, I. 2018. Improving language understanding by generative pre-training. find
- Rajpurkar, P.; Zhang, J.; Lopyrev, K.; and Liang, P. 2016. Squad: 100,000+ questions for machine comprehension of text. In EMNLP, 2383–2392. find our card
- Sakaguchi, K.; Le Bras, R.; Bhagavatula, C.; and Choi, Y. 2020. Winogrande: An adversarial winograd schema challenge at scale. In AAAI. find
- Sap, M.; Rashkin, H.; Chen, D.; Le Bras, R.; and Choi, Y. 2019. Socialiqa: Commonsense reasoning about social interactions. In EMNLP. find our card
- Schoenick, C.; Clark, P.; Tafjord, O.; Turney, P.; and Etzioni, O. 2016. Moving beyond the turing test with the allen ai science challenge. Communications of the ACM. find
- Tellex, S.; Kollar, T.; Dickerson, S.; Walter, M. R.; Banerjee, A. G.; Teller, S.; and Roy, N. 2011. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the National Conference on Artificial Intelligence. find
- Thomason, J.; Sinapov, J.; Svetlik, M.; Stone, P.; and Mooney, R. J. 2016. Learning Multi-Modal Grounded Linguistic Semantics by Playing ”I Spy”. In IJCAI, 3477–3483. find
- Tjong Kim Sang, E. F., and De Meulder, F. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In NAACL, 142–147. find
- Toussaint, M.; Allen, K. R.; Smith, K. A.; and Tenenbaum, J. B. 2018. Differentiable physics and stable modes for tool-use and manipulation planning. In RSS. find
- Wu, J.; Lu, E.; Kohli, P.; Freeman, B.; and Tenenbaum, J. 2017. Learning to see physics via visual de-animation. In Guyon, I.; Luxburg, U. V.; Bengio, S.; Wallach, H.; Fergus, R.; Vishwanathan, S.; and Garnett, R., eds., NeurIPS. find
- Yatskar, M.; Zettlemoyer, L.; and Farhadi, A. 2016. Situation recognition: Visual semantic role labeling for image understanding. In CVPR. find
- Zellers, R.; Bisk, Y.; Schwartz, R.; and Choi, Y. 2018. SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference. In EMNLP. find
- Zellers, R.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019a. From recognition to cognition: Visual commonsense reasoning. In CVPR. find
- Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019b. HellaSwag: Can a Machine Really Finish Your Sentence? In ACL. find