Elamur Research
☰
home / papers / 2023 / Large Language Models Can Be Easily Distracted by Irrelevant Context

Large Language Models Can Be Easily Distracted by Irrelevant Context

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2302.00093

Summary

Most reasoning benchmarks for large language models include only the facts needed to reach the answer. Real tasks are different, because the useful details are mixed with noise. This paper measures how easily prompted LLMs are thrown off by that kind of noise. The authors built GSM-IC from 100 GSM8K problems that at least one prompting method already solves. To each problem they added one template-generated sentence that does not change the correct answer. The added sentence varies in topic (in-topic or off-topic), in whether it reuses role names from the problem, and in the range of numbers it contains. Using Codex (code-davinci-002) and GPT-3.5 (text-davinci-003), they tested chain-of-thought, zero-shot CoT, least-to-most and program-based prompting. Every method lost substantial accuracy when the distractor was present. Three things partly helped: self-consistency decoding, exemplars that also contain irrelevant details, and a short instruction at the start of the prompt saying the model may skip unneeded information. The study shows a basic weakness in context understanding that high benchmark scores hide, and it releases a public benchmark for tracking this kind of robustness.

Key points

  • GSM-IC dataset: the 100 base problems need 2–7 reasoning steps, and 60 of them need only two. A sampled subset called GSM-IC-4K is used for some analyses. The data is published at github.com/google-research-datasets/GSM-IC (ICML 2023).
  • Fragility: of the base problems that the baseline prompts solved with greedy decoding, at most 18% stayed solved across every type of added irrelevant sentence.
  • Self-consistency: with 20 samples per problem, at least one sample gave the correct answer for 99.7% of GSM-IC problems. Adding irrelevant context to the exemplars, or adding the "ignore irrelevant information" instruction, also improved accuracy steadily. Neither change hurt much on clean GSM8K or SVAMP.
  • What makes a sentence distracting: changing the numbers in the distractor made little difference. Lexical overlap with the original problem, such as shared role names, mattered more. A 4-exemplar CoT prompt scored higher than a 1-exemplar prompt on clean GSM8K dev (66.3 vs 60.3). It was less robust to distractors, especially on problems with more than two steps.
  • DROP (football split): the passages here already contain irrelevant text. The instruction still improved both CoT (67.4→68.9) and least-to-most (73.4→74.4) with code-davinci-002.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Mathematics & statistics

Cited by

423 papers of our library cite it (of the 140673 whose bibliographies we read); 23 of them have a page here, 400 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2302.00093 · arXiv 2302.00093

@misc{shi2023large,
  title = {{Large Language Models Can Be Easily Distracted by Irrelevant Context}},
  author = {Shi, Freda and Chen, Xinyun and Misra, Kanishka and Scales, Nathan and Dohan, David and Chi, Ed and Schärli, Nathanael and Zhou, Denny},
  year = {2023},
  doi = {10.48550/arxiv.2302.00093},
  eprint = {2302.00093},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2302.00093}
}

References

56 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Drozdov, A., Schärli, N., Akyürek, E., Scales, N., Song, X., Chen, X., Bousquet, O., and Zhou, D. Compositional semantic parsing with large language models. arXiv preprint arXiv:2209.15003, 2022. find
  2. Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., and Sutton, C. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. find our card
  3. Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., and Gardner, M. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo… find our card
  4. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. NeurIPS, 2020. find
  5. Chaves, R. P. and Richter, S. N. Look at that! BERT can be easily distracted from paying attention to morphosyntax. In Proceedings of the Society for Computation in Linguistics 2021, pp. 28–38, Online, February 2021. Association for Computational Linguistics. URL https: //aclanthology.org/2021.scil-… find
  6. Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., and Neubig, G. Pal: Program-aided language models. arXiv preprint arXiv:2211.10435, 2022. find
  7. Han, S., Schoelkopf, H., Zhao, Y., Qi, Z., Riddell, M., Benson, L., Sun, L., Zubova, E., Qiao, Y., Burtell, M., et al. Folio: Natural language reasoning with first-order logic. arXiv preprint arXiv:2209.00840, 2022. find our card
  8. Chen, W., Ma, X., Wang, X., and Cohen, W. W. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588, 2022. find our card
  9. Hoyer, W. J., Rebok, G. W., and Sved, S. M. Effects of varying irrelevant information on adult age differences in problem solving. Journal of gerontology, 34(4):553–560, 1979. find
  10. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. find
  11. Jia, R. and Liang, P. Adversarial examples for evaluating reading comprehension systems. arXiv preprint arXiv:1707.07328, 2017. find our card
  12. Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. find
  13. Jones, E. and Steinhardt, J. Capturing failures of large language models via human cognitive biases. arXiv preprint arXiv:2202.12299, 2022. — [arXiv:2202.12299](https://arxiv.org/abs/2202.12299) find
  14. Clark, P., Tafjord, O., and Richardson, K. Transformers as soft reasoners over language. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pp. 3882– 3890, 2021. find our card
  15. Cobbe, K., Kosaraju, V., Bavarian, M., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. URL https://arxiv. org/pdf/2110.14168. find our card
  16. Kassner, N. and Schütze, H. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7811–7818, Online, July 2020. Association for Computational Linguistics. doi: 10.1… find our card
  17. Creswell, A., Shanahan, M., and Higgins, I. Selection-inference: Exploiting large language models for interpretable logical reasoning. arXiv preprint arXiv:2205.09712, 2022. find
  18. Khashabi, D., Khot, T., Sabharwal, A., and Roth, D. Learning what is essential in questions. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pp. 80–89, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.18653/v1/K17-1010.… find
  19. Dohan, D., Xu, W., Lewkowycz, A., Austin, J., Bieber, D., Lopes, R. G., Wu, Y., Michalewski, H., Saurous, R. A., Sohl-Dickstein, J., Murphy, K., and Sutton, C. Language model cascades. arXiv preprint arXiv:2207.10342, 2022. find
  20. Khot, T., Trivedi, H., Finlayson, M., Fu, Y., Richardson, K., Clark, P., and Sabharwal, A. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406, 2022. find
  21. Kim, J., Kim, H. J., Cho, H., Jo, H., Lee, S.-W., Lee, S.-g., Yoo, K. M., and Kim, T. Ground-truth labels matter: A deeper look into input-label demonstrations. arXiv preprint arXiv:2205.12685, 2022. find
  22. Ni, J., Zhu, C., Chen, W., and McAuley, J. Learning to attend on essential terms: An enhanced retriever-reader model for open-domain question answering. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  23. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916, 2022. find our card
  24. Kumar, V., Maheshwary, R., and Pudi, V. Adversarial examples for evaluating math word problem solvers. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 2705–2712, 2021. find
  25. Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., Sutton, C., and Odena, A. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021. find our card
  26. Li, D., Rawat, A. S., Zaheer, M., Wang, X., Lukasik, M., Veit, A., Yu, F., and Kumar, S. Large language models with controllable working memory. arXiv preprint arXiv:2211.05110, 2022. find our card
  27. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022. URL https://arxiv. org/abs/2203.02155. find
  28. Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022. find
  29. Ling, W., Yogatama, D., Dyer, C., and Blunsom, P. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 158–167, Vancouver, Canada, July… find
  30. Pandia, L. and Ettinger, A. Sorting through the noise: Testing robustness of information processing in pre-trained language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1583–1596, Online and Punta Cana, Dominican Republic, 2021. Association f… find
  31. Madaan, A. and Yazdanbakhsh, A. Text and patterns: For effective chain of thought, it takes two to tango. arXiv preprint arXiv:2209.07686, 2022. find
  32. Pasolunghi, M. C., Cornoldi, C., and De Liberto, S. Working memory and intrusions of irrelevant information in a group of specific poor problem solvers. Memory & Cognition, 27:779–790, 1999. find
  33. Marzocchi, G. M., Lucangeli, D., De Meo, T., Fini, F., and Cornoldi, C. The disturbing effect of irrelevant information on arithmetic problem solving in inattentive children. Developmental neuropsychology, 21(1):73–92, 2002. find
  34. Patel, A., Bhattamishra, S., and Goyal, N. Are nlp models really able to solve simple math word problems? In NAACL-HLT, 2021. URL https://aclanthology. org/2021.naacl-main.168.pdf. find our card
  35. Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837, 2022. find our card
  36. Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N. A., and Lewis, M. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350, 2022. find
  37. Misra, K., Rayz, J., and Ettinger, A. COMPS: Conceptual minimal pair sentences for testing robust property knowledge and its inheritance in pre-trained language models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, 2023. find
  38. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 2020. URL https://jmlr.org/papers/ v21/20-074.html. ↗
  39. Morris, J. X., Lifland, E., Yoo, J. Y., Grigsby, J., Jin, D., and Qi, Y. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. arXiv preprint arXiv:2005.05909, 2020. find our card
  40. Ravichander, A., Gardner, M., and Marasović, A. Condaqa: A contrastive reading comprehension dataset for reasoning about negation. arXiv preprint arXiv:2211.00295, 2022. find
  41. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., and Zhou, D. Rationale-augmented ensembles in language models. arXiv preprint arXiv:2207.00747, 2022b. find
  42. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., and Zhou, D. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022c. find our card
  43. Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207, 2021. find our card
  44. Webson, A. and Pavlick, E. Do prompt-based models really understand the meaning of their prompts? arXiv preprint arXiv:2109.01247, 2021. find our card
  45. Shi, F., Fried, D., Ghazvininejad, M., Zettlemoyer, L., and Wang, S. I. Natural language to code translation with execution. In EMNLP, 2022a. find
  46. Shi, F., Suzgun, M., Freitag, M., Wang, X., Srivats, S., Vosoughi, S., Chung, H. W., Tay, Y., Ruder, S., Zhou, D., Das, D., and Wei, J. Language models are multilingual chain-of-thought reasoners. arXiv preprint arXiv:2210.03057, 2022b. URL https://arxiv. org/pdf/2210.03057. find our card
  47. Shi, H., Mao, J., Xiao, T., Jiang, Y., and Sun, J. Learning visually-grounded semantics from contrastive adversarial samples. In Proceedings of the 27th International Conference on Computational Linguistics, pp. 3715–3727, 2018. find
  48. Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021. find
  49. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. In NeurIPS, 2022. URL https://openreview.net/ pdf?id=\_VjQlMeSB\_J. ↗ our card
  50. Weston, J., Bordes, A., Chopra, S., Rush, A. M., Van Merriënboer, B., Joulin, A., and Mikolov, T. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698, 2015. find our card
  51. Sinha, K., Sodhani, S., Dong, J., Pineau, J., and Hamilton, W. L. CLUTRR: A diagnostic benchmark for inductive reasoning from text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMN… find
  52. Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Bousquet, O., Le, Q., and Chi, E. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625, 2022. URL https://arxiv.org/pdf/2205.10625. ↗ our card
  53. Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., and Wei, J. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022. find our card
  54. Tafjord, O., Mishra, B. D., and Clark, P. Proofwriter: Generating implications, proofs, and abductive statements over natural language. arXiv preprint arXiv:2012.13048, 2020. find our card
  55. Wang, B., Xu, C., Wang, S., Gan, Z., Cheng, Y., Gao, J., Awadallah, A. H., and Li, B. Adversarial glue: A multi-task benchmark for robustness evaluation of language models. arXiv preprint arXiv:2111.02840, 2021. find
  56. Wang, B., Min, S., Deng, X., Shen, J., Wu, Y., Zettlemoyer, L., and Sun, H. Towards understanding chain-of-thought prompting: An empirical study of what matters. arXiv preprint arXiv:2212.10001, 2022a. *Figure 5. Base problem distribution of GSM-IC with respect to the number of reasoning steps in th… find our card