Elamur Research⌕
☰
home / topics / Language models & pretraining / page 3

🧠 Language models & pretraining

1,079 papers · page 3 of 12 · 126–225 · the most cited first · all topics

Models that learn a language from raw text, and how they are pretrained, tuned and prompted.

1 2 3 4 … 12

  1. WizardLM: Empowering large pre-trained language models to follow complex instructionsCan Xu, Qingfeng Sun, Kai Zheng, et al.2023 · 729 citations
  2. Learning To Retrieve Prompts for In-Context LearningOhad Rubin, Jonathan Herzig, Jonathan Berant2021 · 717 citations
  3. SGLang: Efficient Execution of Structured Language Model ProgramsLianmin Zheng, Liangsheng Yin, Zhiqiang Xie, et al.2023 · 710 citations
  4. Scalable Modified Kneser-Ney Language Model EstimationKenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, et al.2013 · 709 citations
  5. TaBERT: Pretraining for Joint Understanding of Textual and Tabular DataPengcheng Yin, Graham Neubig, Wen-tau Yih, et al.2020 · 690 citations
  6. Explain Yourself! Leveraging Language Models for Commonsense ReasoningNazneen Fatema Rajani, Bryan McCann, Caiming Xiong, et al.2019 · 690 citations
  7. Automatic Chain of Thought Prompting in Large Language ModelsZhuosheng Zhang, Aston Zhang, Mu Li, et al.2022 · 688 citations
  8. ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All ToolsTeam GLM, Aohan Zeng, Bin Xu, et al.2024 · 687 citations
  9. Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model SizesCheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, et al.2023 · 684 citations
  10. DialoGPT: Large-Scale Generative Pre-training for Conversational Response GenerationYizhe Zhang, Siqi Sun, Michel Galley, et al.2019 · 684 citations
  11. Large Language Models Are Human-Level Prompt EngineersYongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, et al.2022 · 677 citations
  12. Mask-Predict: Parallel Decoding of Conditional Masked Language ModelsMarjan Ghazvininejad, Omer Levy, Yinhan Liu, et al.2019 · 671 citations
  13. Well-Read Students Learn Better: On the Importance of Pre-training Compact ModelsIulia Turc, Ming-Wei Chang, Kenton Lee, et al.2019 · 671 citations
  14. Red Teaming Language Models with Language ModelsEthan Perez, Saffron Huang, Francis Song, et al.2022 · 665 citations
  15. An Improved Error Model for Noisy Channel Spelling CorrectionEric Brill, Robert C. Moore2000 · 661 citations
  16. Jailbroken: How Does LLM Safety Training Fail?Alexander Wei, Nika Haghtalab, Jacob Steinhardt2023 · 659 citations
  17. Instruction Tuning with GPT-4Baolin Peng, Chunyuan Li, Pengcheng He, et al.2023 · 651 citations
  18. HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language ModelsJunyi Li, Xiaoxue Cheng, Wayne Xin Zhao, et al.2023 · 646 citations
  19. CoCa: Contrastive Captioners are Image-Text Foundation ModelsJiahui Yu, Zirui Wang, Vijay Vasudevan, et al.2022 · 646 citations
  20. Persistent Anti-Muslim Bias in Large Language ModelsAbubakar Abid, Maheen Farooqi, James Zou2021 · 640 citations
  21. DAPO: An Open-Source LLM Reinforcement Learning System at ScaleQiying Yu, Zheng Zhang, Ruofei Zhu, et al.2025 · 634 citations
  22. Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language ModelsMuhammad Maaz, Hanoona Rasheed, Salman Khan, et al.2023 · 634 citations
  23. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code GenerationJiawei Liu, Chunqiu Steven Xia, Yuyao Wang, et al.2023 · 632 citations
  24. Is ChatGPT A Good Translator? Yes With GPT-4 As The EngineWenxiang Jiao, Wenxuan Wang, Jen-tse Huang, et al.2023 · 626 citations
  25. LayoutLMv2: Multi-modal Pre-training for Visually-rich Document UnderstandingYang Xu, Yiheng Xu, Tengchao Lv, et al.2021 · 623 citations
  26. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?Carlos E. Jimenez, John Yang, Alexander Wettig, et al.2024 · 617 citations
  27. Advances in Pre-Training Distributed Word RepresentationsTomas Mikolov, Edouard Grave, Piotr Bojanowski, et al.2017 · 614 citations
  28. BloombergGPT: A Large Language Model for FinanceShijie Wu, Ozan Irsoy, Steven Lu, et al.2023 · 608 citations
  29. Regularizing and Optimizing LSTM Language ModelsStephen Merity, Nitish Shirish Keskar, Richard Socher2017 · 605 citations
  30. Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language ModelShaden Smith, Mostofa Patwary, Brandon Norick, et al.2022 · 595 citations
  31. Text Embeddings by Weakly-Supervised Contrastive Pre-trainingLiang Wang, Nan Yang, Xiaolong Huang, et al.2022 · 594 citations
  32. The Flan Collection: Designing Data and Methods for Effective Instruction TuningShayne Longpre, Le Hou, Tu Vu, et al.2023 · 590 citations
  33. Instruction-Following Evaluation for Large Language ModelsJeffrey Zhou, Tianjian Lu, Swaroop Mishra, et al.2023 · 586 citations
  34. A Robustly Optimized BERT Pre-training Approach with Post-trainingLiu Zhuang, Lin Wayne, Shi Ya, et al.2021 · 585 citations
  35. PhoBERT: Pre-trained language models for VietnameseDat Quoc Nguyen, Anh Tuan Nguyen2020 · 571 citations
  36. A Hierarchical Bayesian Language Model Based On Pitman-Yor ProcessesYee Whye Teh2006 · 571 citations
  37. AudioLM: a Language Modeling Approach to Audio GenerationZalan Borsos, Raphael Marinier, Damien Vincent, et al.2022 · 570 citations
  38. Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought PromptingMiles Turpin, Julian Michael, Ethan Perez, et al.2023 · 568 citations
  39. Improving Factuality and Reasoning in Language Models through Multiagent DebateYilun Du, Shuang Li, Antonio Torralba, et al.2023 · 564 citations
  40. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIsYujia Qin, Shihao Liang, Yining Ye, et al.2023 · 564 citations
  41. AGIEval: A Human-Centric Benchmark for Evaluating Foundation ModelsWanjun Zhong, Ruixiang Cui, Yiduo Guo, et al.2024 · 563 citations
  42. Plug and Play Language Models: A Simple Approach to Controlled Text GenerationSumanth Dathathri, Andrea Madotto, Janice Lan, et al.2019 · 562 citations
  43. Large Language Models Can Self-ImproveJiaxin Huang, Shixiang Gu, Le Hou, et al.2023 · 560 citations
  44. Galactica: A Large Language Model for ScienceRoss Taylor, Marcin Kardas, Guillem Cucurull, et al.2022 · 558 citations
  45. MetaMath: Bootstrap Your Own Mathematical Questions for Large Language ModelsLonghui Yu, Weisen Jiang, Han Shi, et al.2023 · 556 citations
  46. gpt-oss-120b & gpt-oss-20b Model CardOpenAI: Sandhini Agarwal, Lama Ahmad, Jason Ai, et al.2025 · 555 citations
  47. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for CodeNaman Jain, King Han, Alex Gu, et al.2024 · 540 citations
  48. Release Strategies and the Social Impacts of Language ModelsIrene Solaiman, Miles Brundage, Jack Clark, et al.2019 · 538 citations
  49. Jailbreaking Black Box Large Language Models in Twenty QueriesPatrick Chao, Alexander Robey, Edgar Dobriban, et al.2023 · 533 citations
  50. Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-TuningArmen Aghajanyan, Sonal Gupta, Luke Zettlemoyer2021 · 530 citations
  51. How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source SuitesZhe Chen, Weiyun Wang, Hao Tian, et al.2024 · 529 citations
  52. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language ModelsChaoyou Fu, Peixian Chen, Yunhang Shen, et al.2023 · 529 citations
  53. Gorilla: Large Language Model Connected with Massive APIsShishir G. Patil, Tianjun Zhang, Xin Wang, et al.2023 · 527 citations
  54. Large Language Models Can Be Easily Distracted by Irrelevant ContextFreda Shi, Xinyun Chen, Kanishka Misra, et al.2023 · 525 citations
  55. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model ParametersCharlie Snell, Jaehoon Lee, Kelvin Xu, et al.2024 · 524 citations
  56. Instruction Tuning for Large Language Models: A SurveyShengyu Zhang, Linfeng Dong, Xiaoya Li, et al.2023 · 515 citations
  57. Baichuan 2: Open Large-scale Language ModelsAiyuan Yang, Bin Xiao, Bingning Wang, et al.2023 · 512 citations
  58. OpenAssistant Conversations — Democratizing Large Language Model AlignmentAndreas Köpf, Yannic Kilcher, Dimitri von Rütte, et al.2023 · 505 citations
  59. Sentiment Analysis in the Era of Large Language Models: A Reality CheckWenxuan Zhang, Yue Deng, Bing Liu, et al.2023 · 505 citations
  60. Few-shot Learning with Multilingual Language ModelsXi Victoria Lin, Todor Mihaylov, Mikel Artetxe, et al.2021 · 496 citations
  61. Language Models are Multilingual Chain-of-Thought ReasonersFreda Shi, Mirac Suzgun, Markus Freitag, et al.2022 · 490 citations
  62. Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human AnnotationsPeiyi Wang, Lei Li, Zhihong Shao, et al.2024 · 482 citations
  63. REPLUG: Retrieval-Augmented Black-Box Language ModelsWeijia Shi, Sewon Min, Michihiro Yasunaga, et al.2023 · 482 citations
  64. Enabling Large Language Models to Generate Text with CitationsTianyu Gao, Howard Yen, Jiatong Yu, et al.2023 · 477 citations
  65. GLaM: Efficient Scaling of Language Models with Mixture-of-ExpertsNan Du, Yanping Huang, Andrew M. Dai, et al.2021 · 474 citations
  66. Inference-Time Intervention: Eliciting Truthful Answers from a Language ModelKenneth Li, Oam Patel, Fernanda Viegas, et al.2023 · 473 citations
  67. Large Language Models: A SurveyShervin Minaee, Tomas Mikolov, Narjes Nikzad, et al.2024 · 467 citations
  68. Evaluating Large Language Models in Theory of Mind TasksMichal Kosinski2023 · 465 citations
  69. Masked Autoencoders Are Scalable Vision LearnersKaiming He, Xinlei Chen, Saining Xie, et al.2021 · 463 citations
  70. Text-to-SQL Empowered by Large Language Models: A Benchmark EvaluationDawei Gao, Haibin Wang, Yaliang Li, et al.2023 · 461 citations
  71. SimVLM: Simple Visual Language Model Pretraining with Weak SupervisionZirui Wang, Jiahui Yu, Adams Wei Yu, et al.2021 · 459 citations
  72. Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons LearnedDeep Ganguli, Liane Lovitt, Jackson Kernion, et al.2022 · 456 citations
  73. LinkBERT: Pretraining Language Models with Document LinksMichihiro Yasunaga, Jure Leskovec, Percy Liang2022 · 452 citations
  74. Forest Rescoring: Faster Decoding with Integrated Language ModelsLiang Huang, David Chiang2007 · 452 citations
  75. Show your work: Scratchpads for intermediate computation with language models2021 · 448 citations
  76. LLM-Pruner: On the Structural Pruning of Large Language ModelsXinyin Ma, Gongfan Fang, Xinchao Wang2023 · 447 citations
  77. Reading Level Assessment Using Support Vector Machines and Statistical Language ModelsSarah Schwarm, Mari Ostendorf2005 · 447 citations
  78. HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document SummarizationXingxing Zhang, Furu Wei, Ming Zhou2019 · 446 citations
  79. FinGPT: Open-Source Financial Large Language ModelsHongyang Yang, Xiao-Yang Liu, Christina Dan Wang2023 · 445 citations
  80. How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language ModelsPhillip Rust, Jonas Pfeiffer, Ivan Vulić, et al.2021 · 444 citations
  81. Masked Language Model ScoringJulian Salazar, Davis Liang, Toan Q. Nguyen, et al.2020 · 444 citations
  82. MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual ContextsPan Lu, Hritik Bansal, Tony Xia, et al.2023 · 441 citations
  83. Temporal Analysis of Language through Neural Language ModelsYoon Kim, Yi-I Chiu, Kentaro Hanaki, et al.2014 · 439 citations
  84. GPT-NER: Named Entity Recognition via Large Language ModelsShuhe Wang, Xiaofei Sun, Xiaoya Li, et al.2025 · 436 citations
  85. The Falcon Series of Open Language ModelsEbtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, et al.2023 · 436 citations
  86. Self-Alignment Pretraining for Biomedical Entity RepresentationsFangyu Liu, Ehsan Shareghi, Zaiqiao Meng, et al.2021 · 432 citations
  87. oLMpics — On what Language Model Pre-training CapturesAlon Talmor, Yanai Elazar, Yoav Goldberg, et al.2019 · 432 citations
  88. Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot FlyNora Kassner, Hinrich Schütze2020 · 431 citations
  89. Analyzing the Structure of Attention in a Transformer Language ModelJesse Vig, Yonatan Belinkov2019 · 430 citations
  90. C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation ModelsYuzhen Huang, Yuzhuo Bai, Zhihao Zhu, et al.2023 · 427 citations
  91. The Internal State of an LLM Knows When It's LyingAmos Azaria, Tom Mitchell2023 · 425 citations
  92. Probabilistic Top-Down Parsing and Language ModelingBrian Roark2001 · 424 citations
  93. Security and Privacy Challenges of Large Language Models: A SurveyBadhan Chandra Das, M. Hadi Amini, Yanzhao Wu2024 · 422 citations
  94. Ethical and social risks of harm from Language ModelsLaura Weidinger, John Mellor, Maribeth Rauh, et al.2021 · 421 citations
  95. Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLsJinyang Li, Binyuan Hui, Ge Qu, et al.2023 · 419 citations
  96. Empower Sequence Labeling with Task-Aware Neural Language ModelLiyuan Liu, Jingbo Shang, Frank F. Xu, et al.2017 · 419 citations
  97. ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual LearningViet Dac Lai, Nghia Ngo, Amir Pouran Ben Veyseh, et al.2023 · 418 citations
  98. mPLUG-Owl: Modularization Empowers Large Language Models with MultimodalityQinghao Ye, Haiyang Xu, Guohai Xu, et al.2023 · 410 citations
  99. InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic TasksZhe Chen, Jiannan Wu, Wenhai Wang, et al.2023 · 409 citations
  100. MuRIL: Multilingual Representations for Indian LanguagesSimran Khanuja, Diksha Bansal, Sarvesh Mehtani, et al.2021 · 408 citations

1 2 3 4 … 12

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/topics/language-models/page/3/