Elamur Research⌕
☰
home / papers / 2024 / Pixtral 12B

Pixtral 12B

Paper · 2024 · Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna et al.

Summary

The paper presents Pixtral 12B, an open-weight vision-language model from Mistral AI, released under Apache 2.0. Its goal is to handle both photographs and documents well while keeping text-only skills intact. Many open multimodal models lose some text ability; Pixtral does not. The model combines the Mistral Nemo 12B decoder with Pixtral-ViT, a new 400M-parameter vision encoder trained from scratch. The encoder uses RoPE-2D positional encoding, so it can read images at their original resolution and aspect ratio. Image tokens and text tokens share one 128K-token context, which allows multi-image, multi-turn dialogue.

Pixtral beats open models of similar size, Llama-3.2 11B and Qwen2-VL 7B, on multimodal benchmarks. It also surpasses the much larger Llama-3.2 90B and remains strong on text tasks. The authors found that evaluation setups were inconsistent across papers. To fix this, they propose explicit-format prompts and flexible answer parsing, and they release their evaluation code. They also introduce MM-MT-Bench, a benchmark for practical assistant-style use. Together, these resources make comparisons between multimodal models fairer and closer to real-world use.

Key points

  • Architecture: a 12B decoder (40 layers, dim 5120) is joined to a 24-layer, 400M-parameter Pixtral-ViT encoder through a two-layer GeLU adapter. The encoder adds four changes:
    • [IMAGE BREAK] and [IMAGE END] tokens to mark image rows and endings,
    • gated FFN layers,
    • sequence packing with block-diagonal attention masks,
    • RoPE-2D in place of learned absolute position embeddings.
  • Results: Pixtral is about 7× smaller than Llama-3.2 90B yet beats it on multimodal tasks. It also outperforms the closed models Claude-3 Haiku and Gemini-1.5 Flash 8B.
  • Ranking: in October 2024, it was the top-ranked Apache 2.0 model on the LMSys Vision leaderboard.
  • MM-MT-Bench: the benchmark has 92 conversations, 69 of them single-turn. Images fall into five categories: charts (21), tables (19), PDF pages (24), diagrams (20) and miscellaneous (8). An LLM judge scores each answer from 1 to 10. Scores correlate with LMSys Vision ELO at a Pearson coefficient of 0.91.
  • Evaluation findings: default benchmark prompts are often vague, and exact-match scoring penalizes correct answers in a different format (e.g., "6.0" vs "6"). The authors release the evaluation code as mistral-evals.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 811 works

Cite

Cite as Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amelie Heliou, Paul Jacob, Albert Q. Jiang, Kartik Khandelwal, Timothee Lacroix, Guillaume Lample, Diego Las Casas, Thibaut Lavril, Teven Le Scao, Andy Lo, William Marshall, Louis Martin, Arthur Mensch, Pavankumar Muddireddy, Valera Nemychnikova, Marie Pellat, Patrick Von Platen, Nikhil Raghuraman, Baptiste Roziere, Alexandre Sablayrolles, Lucile Saulnier, Romain Sauvestre, Wendy Shang, Roman Soletskyi, Lawrence Stewart, Pierre Stock, Joachim Studnia, Sandeep Subramanian, Sagar Vaze, Thomas Wang and Sophia Yang. 2024. Pixtral 12B. arXiv:2410.07073.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2410.07073 · arXiv 2410.07073

@misc{agrawal2024pixtral,
  title = {{Pixtral 12B}},
  author = {Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin {De Monicault}, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amelie Heliou, Paul Jacob, Albert Q. Jiang, Kartik Khandelwal, Timothee Lacroix, Guillaume Lample, Diego Las Casas, Thibaut Lavril, Teven {Le Scao}, Andy Lo, William Marshall, Louis Martin, Arthur Mensch, Pavankumar Muddireddy, Valera Nemychnikova, Marie Pellat, Patrick {Von Platen}, Nikhil Raghuraman, Baptiste Roziere, Alexandre Sablayrolles, Lucile Saulnier, Romain Sauvestre, Wendy Shang, Roman Soletskyi, Lawrence Stewart, Pierre Stock, Joachim Studnia, Sandeep Subramanian, Sagar Vaze, Thomas Wang and Sophia Yang},
  year = {2024},
  doi = {10.48550/arxiv.2410.07073},
  eprint = {2410.07073},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.07073}
}

References

26 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic (2024). //www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_ Card_Claude_3.pdf. find
  2. Bavishi, R., Elsen, E., Hawthorne, C., Nye, M., Odena, A., Somani, A., and Taşırlar, S. (2023). Fuyu-8b: A multimodal architecture for ai agents. find
  3. Dehghani, M., Mustafa, B., Djolonga, J., Heek, J., Minderer, M., Caron, M., Steiner, A., Puigcerver, J., Geirhos, R., Alabdulmohsin, I. M., et al. (2024). Patch n’pack: Navit, a vision transformer for any aspect ratio and resolution. Advances in Neural Information Processing Systems, 36. find
  4. Deitke, M., Clark, C., Lee, S., Tripathi, R., Yang, Y., Park, J. S., Salehi, M., Muennighoff, N., Lo, K., Soldaini, L., et al. (2024). Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146.
  5. Dosovitskiy, A. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.
  6. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. (2024). The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  7. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. (2021). Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874.
  8. Hendrycks, D. and Gimpel, K. (2016). Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415. find
  9. Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Li, Y., Liu, Z., and Li, C. (2024). Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326.
  10. Li, X., Wang, Z., and Xie, C. (2023). An inverse scaling law for clip training. In NeurIPS. find
  11. Li, Y. and Harada, T. (2022). Lepard: Learning partial point cloud matching in rigid and deformable scenes. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5554–5564. find
  12. Liu, H., Li, C., Li, Y., and Lee, Y. J. (2024a). Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306.
  13. Liu, H., Li, C., Wu, Q., and Lee, Y. J. (2024b). Visual instruction tuning. Advances in neural information processing systems, 36.
  14. Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. (2023). Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255.
  15. MistralAI (2024). Mistral NeMo 12B. https://mistral.ai/news/mistral-nemo/. ↗
  16. OpenAI, R. et al. (2023). Gpt-4 technical report. ArXiv, 2303:08774.
  17. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021). Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR.
  18. Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lillicrap, T., Alayrac, J.-b., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., et al. (2024). Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.
  19. Shazeer, N. (2020). Glu variants improve transformer. arXiv preprint arXiv:2002.05202. find
  20. Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. (2024). Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063.
  21. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023). Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  22. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.
  23. Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. (2024). Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.
  24. Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. (2023). Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arxiv.
  25. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. (2023). Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.
  26. Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., and Duan, N. (2023). Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/agrawal-2024-pixtral-12b/