Elamur Research⌕
☰
home / papers / 2024 / Nemotron-4 340B Technical Report

Nemotron-4 340B Technical Report

Summary

Nemotron-4 340B is NVIDIA's effort to give the community a very large, openly licensed model family that can also produce synthetic training data. They release three checkpoints under the NVIDIA Open Model License: Nemotron-4-340B-Base, Nemotron-4-340B-Instruct and Nemotron-4-340B-Reward. The base model is a decoder-only Transformer that uses GQA, RoPE and squared ReLU. It was pretrained on 9T tokens of English, multilingual and code text. In the last 1T tokens, the mix shifted toward higher-quality sources and added some question-answering data. Alignment used SFT followed by preference tuning with DPO and RLHF. A multi-attribute regression reward model, trained on the human-labelled HelpSteer2 set, guided this step. Almost all alignment data was machine-generated, and the generation pipeline is released as well. It covers prompt creation, responses, dialogues, filtering and ranking. The base model matches or beats Llama-3 70B, Mixtral 8x22B and Qwen-2 72B on most reasoning benchmarks, and the reward model placed first on RewardBench. The work shows that a permissively licensed model can produce the alignment data it needs and can also generate training data for smaller models.

Key points

  • Pretraining used 9T tokens: 70% English, 15% multilingual (53 natural languages) and 15% code (43 programming languages). It ran as 8T tokens of main pretraining plus 1T tokens of continued training.
  • The model has about 9.4B embedding and 331.6B non-embedding parameters. Training ran on 768 DGX H100 nodes (up to 6,144 GPUs) at roughly 41–42% MFU. In FP8, the model fits on a single 8-GPU DGX H100.
  • Nemotron-4-340B-Base scores ARC-c 94.28, Winogrande 89.50, Hellaswag 90.53, BBH 85.44, MMLU 81.10 and HumanEval 57.32.
  • Over 98% of the alignment data is synthetic. HelpSteer2 (about 10k human preference annotations) is released to train the reward model, which beat GPT-4o-0513 and Gemini 1.5 Pro-0514 on RewardBench.
  • Safety testing used AEGIS and the Garak scanner. Overall safety was comparable to Llama-3-70B-Instruct, but weak spots remain: jailbreaks, some malware requests and adversarial hallucination.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Cited by

95 papers of our library cite it (of the 140884 whose bibliographies we read); 11 of them have a page here, 84 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2406.11704 · arXiv 2406.11704

@misc{adler2024nemotron,
  title = {{Nemotron-4 340B Technical Report}},
  author = {Adler, Nvidia: Bo and Agarwal, Niket and Aithal, Ashwath and Anh, Dong H. and Bhattacharya, Pallab and Brundyn, Annika and Casper, Jared and Catanzaro, Bryan and Clay, Sharon and Cohen, Jonathan and Das, Sirshak and Dattagupta, Ayush and Delalleau, Olivier and Derczynski, Leon and Dong, Yi and Egert, Daniel and Evans, Ellie and Ficek, Aleksander and Fridman, Denys and Ghosh, Shaona and Ginsburg, Boris and Gitman, Igor and Grzegorzek, Tomasz and Hero, Robert and Huang, Jining and Jawa, Vibhu and Jennings, Joseph and Jhunjhunwala, Aastha and Kamalu, John and Khan, Sadaf and Kuchaiev, Oleksii and LeGresley, Patrick and Li, Hui and Liu, Jiwei and Liu, Zihan and Long, Eileen and Mahabaleshwarkar, Ameya Sunil and Majumdar, Somshubra and Maki, James and Martinez, Miguel and Melo, Maer Rodrigues de and Moshkov, Ivan and Narayanan, Deepak and Narenthiran, Sean and Navarro, Jesus and Nguyen, Phong and Nitski, Osvald and Noroozi, Vahid and Nutheti, Guruprasad and Parisien, Christopher and Parmar, Jupinder and Patwary, Mostofa and Pawelec, Krzysztof and Ping, Wei and Prabhumoye, Shrimai and Roy, Rajarshi and Saar, Trisha and Sabavat, Vasanth Rao Naik and Satheesh, Sanjeev and Scowcroft, Jane Polak and Sewall, Jason and Shamis, Pavel and Shen, Gerald and Shoeybi, Mohammad and Sizer, Dave and Smelyanskiy, Misha and Soares, Felipe and Sreedhar, Makesh Narsimhan and Su, Dan and Subramanian, Sandeep and Sun, Shengyang and Toshniwal, Shubham and Wang, Hao and Wang, Zhilin and You, Jiaxuan and Zeng, Jiaqi and Zhang, Jimmy and Zhang, Jing and Zhang, Vivienne and Zhang, Yian and Zhu, Chen},
  year = {2024},
  doi = {10.48550/arxiv.2406.11704},
  eprint = {2406.11704},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.11704}
}

References

72 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. NVLink and NVSwitch. https://www.nvidia.com/en-us/data-center/nvlink/. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. arXiv preprint arXiv:2305.13245, 2023. ↗ our card
  2. Allen AI. Reward bench leaderboard. https://huggingface.co/spaces/allenai/reward-bench, 2024. ↗
  3. Importantly, the safety model can make both false positive and false negative errors. Future work will include human ground truth labels to quantify the false prediction rate. * indicates equal contribution † indicates work done while at NVIDIA AI Anthropic. The claude 3 model family: Opus, sonnet,… find
  4. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program Synthesis with Large Language Models, 2021. find our card
  5. Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447–4455. PMLR, 2024… find our card
  6. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. find
  7. Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390, 2023. find our card
  8. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… find our card
  9. Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, et al. Finqa: A dataset of numerical reasoning over financial data. arXiv preprint arXiv:2109.00122, 2021b. find our card
  10. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling Language Modeling with Pathways. arXiv preprint arXiv:2204.02311, 2022. find
  11. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think You have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv preprint arXiv:1803.05457, 2018. find
  12. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems. CoRR, abs/2110.14168, 2021. URL https://arxiv.org/abs/2110.14168. ↗ our card
  13. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023. find
  14. Leon Derczynski, Erick Galinkin, Jeffrey Martin, Subho Majumdar, and Nanna Inie. garak: A Framework for Security Probing Large Language Models. arXiv preprint arXiv:2406.11036, 2024. — [arXiv:2406.11036](https://arxiv.org/abs/2406.11036) find
  15. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233, 2023. find our card
  16. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024. find
  17. Adam Fisch, Jacob Eisenstein, Vicky Zayats, Alekh Agarwal, Ahmad Beirami, Chirag Nagpal, Pete Shaw, and Jonathan Berant. Robust preference optimization through reward model distillation. arXiv preprint arXiv:2405.19316, 2024. — [arXiv:2405.19316](https://arxiv.org/abs/2405.19316) find
  18. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A Framework for Few-shot Language Model Evaluation, September… — [doi:10.5281/zenodo.5371628](https://doi.org/10.5281/zenodo.5371628) find
  19. Shaona Ghosh, Prasoon Varshney, Erick Galinkin, and Christopher Parisien. AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Experts. arXiv preprint arXiv:2404.05993, 2024. — [arXiv:2404.05993](https://arxiv.org/abs/2404.05993) find
  20. Glaive AI. glaive-function-calling-v2. glaive-function-calling-v2, 2023. https://huggingface.co/datasets/glaiveai/ ↗
  21. Loubna Ben Allal Anton Lozhkov Colin Raffel Leandro Werra Thomas Wolf Guilherme Penedo, Hynek Kydlı́ček. Fineweb: decanting the web for the finest text data at scale. https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1, 2024. ↗
  22. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding. arXiv preprint arXiv:2009.03300, 2020. find our card
  23. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. find our card
  24. Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674, 2023. find our card
  25. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. find
  26. Ankur Joshi, Saket Kale, Satish Chandel, and D Kumar Pal. Likert scale: Explored and explained. British journal of applied science & technology, 7(4):396–403, 2015. find
  27. Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. Reducing Activation Recomputation in Large Transformer Models, 2022. find our card
  28. Taku Kudo and John Richardson. Sentencepiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing. arXiv preprint arXiv:1808.06226, 2018. find our card
  29. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787, 2024. find
  30. Ariel N Lee, Cole J Hunter, and Nataniel Ruiz. Platypus: Quick, cheap, and powerful refinement of llms. arXiv preprint arXiv:2308.07317, 2023. find
  31. Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for ”mind” exploration of large language model society, 2023. find our card
  32. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The arena-hard pipeline, april 2024. URL https://lmsys.org/blog/2024-04-19-arena-hard. ↗
  33. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.20050, 2023. find our card
  34. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum? id=1qvx61… ↗ our card
  35. Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, M Shoeybi, and B Catanzaro. Chatqa: Surpassing gpt-4 on conversational qa and rag. arXiv preprint arXiv:2401.10225, 2024. find
  36. Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568, 2023. find
  37. Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. Rephrasing the web: A recipe for compute and data-efficient language modeling, 2024. find
  38. MetaAI. Introducing meta llama 3: The most capable openly available llm to date. https://ai.meta.com/blog/meta-llama-3/, 2024. ↗
  39. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018. find
  40. Mistral-AI-Team. Mistral large. https://mistral.ai/news/mistral-large, 2024a. ↗
  41. Mistral-AI-Team. Mistral 8x22b. https://mistral.ai/news/mixtral-8x22b, 2024b. ↗
  42. Eric Mitchell. A note on dpo with noisy preferences & relationship to ipo, 2023. find
  43. Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training on GPU clusters using Megatron-LM. In Proceedings of the International… find
  44. NVIDIA. H100 Tensor Core GPU Architecture Overview, 2022. find
  45. OpenAI. Gpt-4-1106-preview. gpt-4-turbo-and-gpt-4, 2023. https://platform.openai.com/docs/models/ ↗
  46. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022. find
  47. Gaurav Pandey, Yatin Nandwani, Tahira Naseem, Mayank Mishra, Guangxuan Xu, Dinesh Raghu, Sachindra Joshi, Asim Munawar, and Ramón Fernandez Astudillo. Brain: Bayesian reward-conditioned amortized inference for natural language generation from feedback. arXiv preprint arXiv:2402.02479, 2024. — [arXiv:2402.02479](https://arxiv.org/abs/2402.02479) find
  48. Jupinder Parmar, Shrimai Prabhumoye, Joseph Jennings, Mostofa Patwary, Sandeep Subramanian, Dan Su, Chen Zhu, Deepak Narayanan, Aastha Jhunjhunwala, Ayush Dattagupta, Vibhu Jawa, Jiwei Liu, Ameya Mahabaleshwarkar, Osvald Nitski, Annika Brundyn, James Maki, Miguel Martinez, Jiaxuan You, John Kamalu,… find
  49. Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. arXiv preprint arXiv:1508.00305, 2015. find
  50. Qwen-Team. Hello qwen2. https://qwenlm.github.io/blog/qwen2, 2024. ↗
  51. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024. find
  52. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020. find
  53. Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.03715, 2024. find
  54. RyokoAI. RyokoAI/ShareGPT52K. https://huggingface.co/datasets/RyokoAI/ShareGPT52K, 2023. ↗
  55. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WINOGRANDE: An Adversarial Winograd Schema Challenge at Scale. In AAAI, 2020. find
  56. Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John J Nay, Kshitij Gupta, and Aran Komatsuzaki. Arb: Advanced reasoning benchmark for large language models. arXiv preprint arXiv:2307.13692, 2023. find
  57. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training Multi-Billion Parameter Language Models using Model Parallelism. arXiv preprint arXiv:1909.08053, 2019. find
  58. Makesh Narsimhan Sreedhar, Traian Rebedea, Shaona Ghosh, and Christopher Parisien. Canttalkaboutthis: Aligning language models to stay on topic in dialogues, 2024. find
  59. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced Transformer with Rotary Position Embedding. arXiv preprint arXiv:2104.09864, 2021. find
  60. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them, 2022. find our card
  61. Evan Frick Lisa Dunlap Banghua Zhu Joseph E. Gonzalez Ion Stoica Tianle Li*, Wei-Lin Chiang*. From live data to high-quality benchmarks: The arena-hard pipeline, April 2024. URL https://lmsys.org/ blog/2024-04-19-arena-hard/. ↗
  62. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open Foundation and Fine-tuned Chat Models. arXiv preprint arXiv:2307.09288, 2023. find
  63. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1706.03762, 2017. URL http://arxiv.org/abs/1706.03762. ↗
  64. Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scientific problem-solving abilities of large language models. arXiv preprint arXiv:2307.10635, 2023a. find our card
  65. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560, 2022. find
  66. Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, et al. Helpsteer: Multi-attribute helpfulness dataset for steerlm. arXiv preprint arXiv:2311.09528, 2023b. find
  67. Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models, 2024. find
  68. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a Machine Really Finish Your Sentence? In ACL, 2019. find
  69. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, et al. Lmsys-chat-1m: A large-scale real-world llm conversation dataset. arXiv preprint arXiv:2309.11998, 2023. find our card
  70. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024a. find
  71. Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeinterpreter: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402.14658, 2024b. find
  72. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/adler-2024-nemotron-4-340b-technical-report/