Elamur Research⌕
☰
home / papers / 2023 / Fool Your (Vision and) Language Model With Embarrassingly Simple Permutations

Fool Your (Vision and) Language Model With Embarrassingly Simple Permutations

Paper · 2023 · Yongshuo Zong, Tingyang Yu, Ruchika Chavhan et al.

Summary

Benchmarks built on multiple-choice question answering (MCQA) are widely used to assess text-only LLMs and their vision-language counterparts, VLLMs. This paper asks whether those scores hold up when only the order of answer options changes. The authors propose a simple adversarial attack. For each question, they cycle through orderings of the options and stop at the first one that produces a wrong answer. The models evaluated include LLaMA-2, Vicuna, WizardLM, InternLM, Falcon, MPT and GPT-3.5-turbo. On the vision side, they test InstructBLIP, OpenFlamingo, Otter, LLaVA, LLaVA-v1.5, Emu2-Chat, Limber and mPLUG-Owl. The benchmarks are five text and four multimodal MCQA datasets. Accuracy falls sharply under the attack and often ends up below random guessing. This happens even when distractor options are removed, and the effect is stronger than plain position bias can explain. Majority voting, calibration and PriDe do not fully fix the problem. Fine-tuning on permuted options helps only partly. The results suggest that MCQA benchmark scores may overstate how robust these models really are.

Key points

  • Text datasets: MMLU, ARC-c, BoolQ, SocialiQA, MedMCQA. Vision-language datasets: ScienceQA, A-OKVQA, MMBench, SEED-Bench. Code is available at github.com/ys-zong/FoolyourVLLMs (ICML 2024).
  • On MMLU, Llama2-13B drops from 52.22% to 18.33% accuracy after permutation, a loss of 33.89 points that puts it below chance.
  • In the VLLM experiments, more than half of 48 settings fall below chance, with accuracy drops of roughly 20–45%. Larger models within a family tend to be somewhat more resilient. On MedMCQA, only GPT-3.5-turbo stays above chance.
  • Defenses on MMLU give mixed results:
    • Contextual calibration fails.
    • PriDe does not help against full permutation.
    • Majority vote and max-confidence recover much of the lost accuracy, but at the cost of extra inference over many orderings.
  • LoRA fine-tuning of Llama2-7B with permutation-augmented data raises permuted accuracy on ARC-Challenge from 7.98% (zero-shot) to 47.73%. A gap from the unpermuted score still remains.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 4,145 worksVision & language · 827 works

Cite

Cite as Yongshuo Zong, Tingyang Yu, Ruchika Chavhan, Bingchen Zhao and Timothy Hospedales. 2023. Fool Your (Vision and) Language Model With Embarrassingly Simple Permutations. arXiv:2310.01651.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.01651 · arXiv 2310.01651

@misc{zong2023fool,
  title = {{Fool Your (Vision and) Language Model With Embarrassingly Simple Permutations}},
  author = {Yongshuo Zong, Tingyang Yu, Ruchika Chavhan, Bingchen Zhao and Timothy Hospedales},
  year = {2023},
  doi = {10.48550/arxiv.2310.01651},
  eprint = {2310.01651},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.01651}
}

References

47 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., and Wichmann, F. A. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020. find
  2. Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: a visual language model for few-shot learning. NeurIPS, 2022.
  3. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
  4. Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023.
  5. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. ICLR, 2022.
  6. Li, B., Wang, R., Wang, G., Ge, Y., Ge, Y., and Shan, Y. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023a.
  7. Awadalla, A., Gao, I., Gardner, J., Hessel, J., Hanafy, Y., Zhu, W., Marathe, K., Bitton, Y., Gadre, S., Sagawa, S., Jitsev, J., Kornblith, S., Koh, P. W., Ilharco, G., Wortsman, M., and Schmidt, L. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXi…
  8. Li, B., Zhang, Y., Chen, L., Wang, J., Yang, J., and Liu, Z. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726, 2023b.
  9. Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. ICML, 2023c.
  10. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. NeurIPS, 2020.
  11. Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022. find
  12. Chen, L., Davis, J. Q., Hanin, B., Bailis, P., Stoica, I., Zaharia, M., and Zou, J. Are more llm calls all you need? towards scaling laws of compound inference systems. arXiv, 2024. find
  13. Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023a.
  14. Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna.lmsys. org (accessed 14 April 2023), 2023. ↗
  15. Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. arXiv:2304.08485, 2023b.
  16. Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., and Chen, W. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06804, 2021.
  17. Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019.
  18. Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023c.
  19. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved Liu, Y., Yao, Y., Ton, J.-F., Zhang, X., Cheng, R. G. H., Klochkov, Y., Taufiq, M. F., and Li, H. Trustworthy llms: a survey and guideline for evaluating large language models’ alignment… — Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment
  20. Schwenk, D., Khandelwal, A., Clark, C., Marino, K., and Mottaghi, R. A-okvqa: A benchmark for visual question answering using world knowledge. In European Conference on Computer Vision, pp. 146–162. Springer, 2022.
  21. Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521, 2022. find
  22. Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E. H., Schärli, N., and Zhou, D. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pp. 31210– 31227. PMLR, 2023.
  23. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
  24. Sun, Q., Cui, Y., Zhang, X., Zhang, F., Yu, Q., Luo, Z., Wang, Y., Rao, Y., Liu, J., Huang, T., et al. Generative multimodal models are in-context learners. CVPR, 2024. find
  25. Merullo, J., Castricato, L., Eickhoff, C., and Pavlick, E. Linearly mapping from image to text space. ICLR, 2023. find
  26. Team, I. Internlm: A multilingual language model with progressively enhanced capabilities. https://github.com/InternLM/InternLM, 2023a. ↗
  27. OpenAI. Introducing chatgpt. https://openai.com/ blog/chatgpt, 2023a. ↗
  28. Team, M. N. Introducing mpt-7b: A new standard for open-source, commercially usable llms, 2023b. URL www.mosaicml.com/blog/mpt-7b. find
  29. OpenAI, R. Gpt-4 technical report. arXiv, pp. 2303–08774, 2023b.
  30. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 2022.
  31. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a.
  32. Pal, A., Umapathi, L. K., and Sankarasubbu, M. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on Health, Inference, and Learning. PMLR, 2022. — MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering
  33. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.
  34. Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Cappelli, A., Alobeidli, H., Pannier, B., Almazrouei, E., and Launay, J. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116, 2023. find
  35. Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=1PL1NIMMrw. ↗
  36. Robinson, J. and Wingate, D. Leveraging large language models for multiple choice question answering. In The Eleventh International Conference on Learning Representations, 2023.
  37. Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., and Jiang, D. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023.
  38. Sagawa, S., Raghunathan, A., Koh, P. W., and Liang, P. An investigation of why overparameterization exacerbates spurious correlations. In International Conference on Machine Learning, pp. 8346–8356. PMLR, 2020. find
  39. Ye, Q., Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y., Wang, J., Hu, A., Shi, P., Shi, Y., Jiang, C., Li, C., Xu, Y., Chen, H., Tian, J., Qi, Q., Zhang, J., and Huang, F. mplug-owl: Modularization empowers large language models with multimodality. 2304.14178, 2023.
  40. Sap, M., Rashkin, H., Chen, D., Le Bras, R., and Choi, Y. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2… find
  41. Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., and Chen, E. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549, 2023.
  42. Zhao, Z., Wallace, E., Feng, S., Klein, D., and Singh, S. Calibrate before use: Improving few-shot performance of language models. In ICML, 2021.
  43. Zheng, C., Zhou, H., Meng, F., Zhou, J., and Huang, M. On large language models’ selection bias in multi-choice questions. arXiv preprint arXiv:2309.03882, 2023a. find
  44. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023b.
  45. Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., and Duan, N. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364, 2023.
  46. Zhu, K., Wang, J., Zhou, J., Wang, Z., Chen, H., Wang, Y., Yang, L., Ye, W., Gong, N. Z., Zhang, Y., et al. Promptbench: Towards evaluating the robustness of large language models on adversarial prompts. arXiv preprint arXiv:2306.04528, 2023.
  47. Zong, Y., Mac Aodha, O., and Hospedales, T. Self-supervised multimodal learning: A survey. arXiv, 2023. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zong-2023-fool-vision-language-model-embarrassingly/