Elamur Research⌕
☰
home / topics / Language models & pretraining / page 4

🧠 Language models & pretraining

1,079 papers · page 4 of 12 · 226–325 · the most cited first · all topics

Models that learn a language from raw text, and how they are pretrained, tuned and prompted.

1 2 3 4 5 … 12

  1. UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language ModelsTianbao Xie, Chen Henry Wu, Peng Shi, et al.2022 · 405 citations
  2. Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video AnalysisChaoyou Fu, Yuhan Dai, Yongdong Luo, et al.2024 · 403 citations
  3. True Few-Shot Learning with Language ModelsEthan Perez, Douwe Kiela, Kyunghyun Cho2021 · 403 citations
  4. Yi: Open Foundation Models by 01.AIAlex Young, Bei Chen, Chao Li, et al.2024 · 402 citations
  5. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUYing Sheng, Lianmin Zheng, Binhang Yuan, et al.2023 · 399 citations
  6. LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language ModelsZhiqiang Hu, Lei Wang, Yihuai Lan, et al.2023 · 398 citations
  7. DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-CorrectionMohammadreza Pourreza, Davood Rafiei2023 · 397 citations
  8. Large Language Models Cannot Self-Correct Reasoning YetJie Huang, Xinyun Chen, Swaroop Mishra, et al.2023 · 396 citations
  9. Whose Opinions Do Language Models Reflect?Shibani Santurkar, Esin Durmus, Faisal Ladhak, et al.2023 · 390 citations
  10. Large Language Models as OptimizersChengrun Yang, Xuezhi Wang, Yifeng Lu, et al.2023 · 388 citations
  11. BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelsElad Ben-Zaken, Shauli Ravfogel, Yoav Goldberg2021 · 388 citations
  12. RULER: What's the Real Context Size of Your Long-Context Language Models?Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, et al.2024 · 386 citations
  13. Immediate-Head Parsing for Language ModelsEugene Charniak2001 · 386 citations
  14. German's Next Language ModelBranden Chan, Stefan Schweter, Timo Möller2020 · 385 citations
  15. Query Rewriting in Retrieval-Augmented Large Language ModelsXinbei Ma, Yeyun Gong, Pengcheng He, et al.2023 · 383 citations
  16. Enhancing Chat Language Models by Scaling High-quality Instructional ConversationsNing Ding, Yulin Chen, Bokai Xu, et al.2023 · 382 citations
  17. Gender and Representation Bias in GPT-3 Generated StoriesLi Lucy, David Bamman2021 · 382 citations
  18. A Survey on Model Compression for Large Language ModelsXunyu Zhu, Jian Li, Yong Liu, et al.2024 · 381 citations
  19. Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question ComplexitySoyeong Jeong, Jinheon Baek, Sukmin Cho, et al.2024 · 381 citations
  20. Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation SynergyZhihong Shao, Yeyun Gong, Yelong Shen, et al.2023 · 381 citations
  21. UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-TrainingHangbo Bao, Li Dong, Furu Wei, et al.2020 · 380 citations
  22. Tulu 3: Pushing Frontiers in Open Language Model Post-TrainingNathan Lambert, Jacob Morrison, Valentina Pyatkin, et al.2024 · 379 citations
  23. Emerging Cross-lingual Structure in Pretrained Language ModelsAlexis Conneau, Shijie Wu, Haoran Li, et al.2020 · 378 citations
  24. Tool Learning with Foundation ModelsYujia Qin, Shengding Hu, Yankai Lin, et al.2023 · 376 citations
  25. A Watermark for Large Language ModelsJohn Kirchenbauer, Jonas Geiping, Yuxin Wen, et al.2023 · 374 citations
  26. CogVLM: Visual Expert for Pretrained Language ModelsWeihan Wang, Qingsong Lv, Wenmeng Yu, et al.2023 · 373 citations
  27. A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and ApplicationsPranab Sahoo; Ayush Kumar Singh; Sriparna Saha; Vinija Jain; Samrat Mondal;, Aman Chadha2024 · 371 citations
  28. Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical ReasoningLiangming Pan, Alon Albalak, Xinyi Wang, et al.2023 · 370 citations
  29. Large Language Models Are Reasoning TeachersNamgyu Ho, Laura Schmid, Se-Young Yun2023 · 368 citations
  30. VILA: On Pre-training for Visual Language ModelsJi Lin, Hongxu Yin, Wei Ping, et al.2023 · 367 citations
  31. A Survey on Large Language Models for Code GenerationJuyong Jiang, Fan Wang, Jiasi Shen, et al.2024 · 366 citations
  32. WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-InstructHaipeng Luo, Qingfeng Sun, Can Xu, et al.2023 · 366 citations
  33. Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic CoherenceFederico Bianchi, Silvia Terragni, Dirk Hovy2021 · 365 citations
  34. The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language ModelsGo Inoue, Bashar Alhafni, Nurpeiis Baimukan, et al.2021 · 365 citations
  35. Identifying and Reducing Gender Bias in Word-Level Language ModelsShikha Bordia, Samuel R. Bowman2019 · 363 citations
  36. Augmented Language Models: a SurveyGrégoire Mialon, Roberto Dessì, Maria Lomeli, et al.2023 · 362 citations
  37. TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented DialogueChien-Sheng Wu, Steven C.H. Hoi, Richard Socher, et al.2020 · 360 citations
  38. Orca: Progressive Learning from Complex Explanation Traces of GPT-4Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, et al.2023 · 357 citations
  39. Unnatural Instructions: Tuning Language Models with (Almost) No Human LaborOr Honovich, Thomas Scialom, Omer Levy, et al.2023 · 356 citations
  40. XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and GenerationYaobo Liang, Nan Duan, Yeyun Gong, et al.2020 · 356 citations
  41. Multilingual Constituency Parsing with Self-Attention and Pre-TrainingNikita Kitaev, Steven Cao, Dan Klein2019 · 356 citations
  42. Synthetic QA Corpora Generation with Roundtrip ConsistencyChris Alberti, Daniel Andor, Emily Pitler, et al.2019 · 355 citations
  43. Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated FeedbackBaolin Peng, Michel Galley, Pengcheng He, et al.2023 · 354 citations
  44. Towards Understanding Sycophancy in Language ModelsMrinank Sharma, Meg Tong, Tomasz Korbak, et al.2023 · 354 citations
  45. Towards Mitigating LLM Hallucination via Self ReflectionZiwei Ji, Tiezheng Yu, Yan Xu, et al.2023 · 352 citations
  46. Large Language Models Struggle to Learn Long-Tail KnowledgeNikhil Kandpal, Haikang Deng, Adam Roberts, et al.2022 · 352 citations
  47. BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference DatasetJiaming Ji, Mickel Liu, Juntao Dai, et al.2023 · 351 citations
  48. RARR: Researching and Revising What Language Models Say, Using Language ModelsLuyu Gao, Zhuyun Dai, Panupong Pasupat, et al.2023 · 351 citations
  49. SnapKV: LLM Knows What You are Looking for Before GenerationYuhong Li, Yingbing Huang, Bowen Yang, et al.2024 · 350 citations
  50. TinyLlama: An Open-Source Small Language ModelPeiyuan Zhang, Guangtao Zeng, Tianduo Wang, et al.2024 · 350 citations
  51. SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational AbilitiesDong Zhang, Shimin Li, Xin Zhang, et al.2023 · 346 citations
  52. RobBERT: a Dutch RoBERTa-based Language ModelPieter Delobelle, Thomas Winters, Bettina Berendt2020 · 346 citations
  53. Toxicity in chatgpt: Analyzing persona-assigned language modelsAmeet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, et al.2023 · 344 citations
  54. Complexity-Based Prompting for Multi-Step ReasoningYao Fu, Hao Peng, Ashish Sabharwal, et al.2022 · 343 citations
  55. Symbolic Knowledge Distillation: from General Language Models to Commonsense ModelsPeter West, Chandra Bhagavatula, Jack Hessel, et al.2022 · 343 citations
  56. Teaching Small Language Models to ReasonLucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, et al.2023 · 340 citations
  57. Large Language Models for Information Retrieval: A SurveyYutao Zhu; Huaying Yuan; Shuting Wang; Jiongnan Liu; Wenhan Liu; Chenlong Deng; Haonan Chen; Zheng Liu; Zhicheng Dou;, Ji-Rong Wen2023 · 335 citations
  58. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language ModelsHuiqiang Jiang, Qianhui Wu, Chin-Yew Lin, et al.2023 · 335 citations
  59. UL2: Unifying Language Learning ParadigmsYi Tay, Mostafa Dehghani, Vinh Q. Tran, et al.2022 · 334 citations
  60. DeepSeek-V3.2: Pushing the Frontier of Open Large Language ModelsDeepSeek-AI, Aixin Liu, Aoxue Mei, et al.2025 · 333 citations
  61. AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language ModelsXiaogeng Liu, Nan Xu, Muhao Chen, et al.2023 · 333 citations
  62. Neural Codec Language Models are Zero-Shot Text to Speech SynthesizersChengyi Wang, Sanyuan Chen, Yu Wu, et al.2023 · 333 citations
  63. Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!Xiangyu Qi, Yi Zeng, Tinghao Xie, et al.2023 · 331 citations
  64. TimeLMs: Diachronic Language Models from TwitterDaniel Loureiro, Francesco Barbieri, Leonardo Neves, et al.2022 · 331 citations
  65. Unsupervised Corpus Aware Language Model Pre-training for Dense Passage RetrievalLuyu Gao, Jamie Callan2022 · 328 citations
  66. GraphCodeBERT: Pre-training Code Representations with Data FlowDaya Guo, Shuo Ren, Shuai Lu, et al.2021 · 328 citations
  67. HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language ModelsTianrui Guan, Fuxiao Liu, Xiyang Wu, et al.2023 · 326 citations
  68. Unsupervised Domain Clusters in Pretrained Language ModelsRoee Aharoni, Yoav Goldberg2020 · 326 citations
  69. ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and GenerationYu Sun, Shuohuan Wang, Shikun Feng, et al.2021 · 324 citations
  70. Are Emergent Abilities of Large Language Models a Mirage?Rylan Schaeffer, Brando Miranda, Sanmi Koyejo2023 · 323 citations
  71. LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init AttentionRenrui Zhang, Jiaming Han, Chris Liu, et al.2023 · 322 citations
  72. "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language ModelsXinyue Shen, Zeyuan Chen, Michael Backes, et al.2023 · 320 citations
  73. Probing Pretrained Language Models for Lexical SemanticsIvan Vulić, Edoardo Maria Ponti, Robert Litschko, et al.2020 · 320 citations
  74. Language Is Not All You Need: Aligning Perception with Language ModelsShaohan Huang, Li Dong, Wenhui Wang, et al.2023 · 319 citations
  75. Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive DecodingSicong Leng, Hang Zhang, Guanzheng Chen, et al.2023 · 317 citations
  76. Text and Code Embeddings by Contrastive Pre-TrainingArvind Neelakantan, Tao Xu, Raul Puri, et al.2022 · 316 citations
  77. CRITIC: Large Language Models Can Self-Correct with Tool-Interactive CritiquingZhibin Gou, Zhihong Shao, Yeyun Gong, et al.2023 · 315 citations
  78. Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation ModelsChenfei Wu, Shengming Yin, Weizhen Qi, et al.2023 · 315 citations
  79. LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative FusionDongfu Jiang, Xiang Ren, Bill Yuchen Lin2023 · 314 citations
  80. Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation ModelsYanzhao Zhang, Mingxin Li, Dingkun Long, et al.2025 · 313 citations
  81. Unsupervised Pretraining for Sequence to Sequence LearningPrajit Ramachandran, Peter Liu, Quoc Le2017 · 312 citations
  82. Hallucination is Inevitable: An Innate Limitation of Large Language ModelsZiwei Xu, Sanjay Jain, Mohan Kankanhalli2024 · 309 citations
  83. From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP ModelsShangbin Feng, Chan Young Park, Yuhan Liu, et al.2023 · 305 citations
  84. Discovering Language Model Behaviors with Model-Written EvaluationsEthan Perez, Sam Ringer, Kamilė Lukošiūtė, et al.2022 · 305 citations
  85. How Far Can Camels Go? Exploring the State of Instruction Tuning on Open ResourcesYizhong Wang, Hamish Ivison, Pradeep Dasigi, et al.2023 · 304 citations
  86. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement LearningBowen Jin, Hansi Zeng, Zhenrui Yue, et al.2025 · 303 citations
  87. The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)Zhengyuan Yang, Linjie Li, Kevin Lin, et al.2023 · 303 citations
  88. Large Language Models are Better Reasoners with Self-VerificationYixuan Weng, Minjun Zhu, Fei Xia, et al.2023 · 302 citations
  89. PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial PromptsKaijie Zhu, Jindong Wang, Jiaheng Zhou, et al.2023 · 301 citations
  90. Challenges and Applications of Large Language ModelsJean Kaddour, Joshua Harris, Maximilian Mozes, et al.2023 · 300 citations
  91. Want To Reduce Labeling Cost? GPT-3 Can HelpShuohang Wang, Yang Liu, Yichong Xu, et al.2021 · 298 citations
  92. Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMsShengbang Tong, Zhuang Liu, Yuexiang Zhai, et al.2024 · 297 citations
  93. Chain-of-Verification Reduces Hallucination in Large Language ModelsShehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, et al.2023 · 297 citations
  94. Large Language Models are Effective Text Rankers with Pairwise Ranking PromptingZhen Qin, Rolf Jagerman, Kai Hui, et al.2024 · 296 citations
  95. Multimodal Chain-of-Thought Reasoning in Language ModelsZhuosheng Zhang, Aston Zhang, Mu Li, et al.2023 · 296 citations
  96. BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in BanglaAbhik Bhattacharjee, Tahmid Hasan, Wasi Ahmad, et al.2022 · 296 citations
  97. Predictive power of word surprisal for reading times is a linear function of language model qualityAdam Goodkind, Klinton Bicknell2018 · 294 citations
  98. KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache QuantizationColeman Hooper, Sehoon Kim, Hiva Mohammadzadeh, et al.2024 · 293 citations
  99. FlauBERT: Unsupervised Language Model Pre-training for FrenchHang Le, Loïc Vial, Jibril Frej, et al.2019 · 293 citations
  100. Text Classification via Large Language ModelsXiaofei Sun, Xiaoya Li, Jiwei Li, et al.2023 · 292 citations

1 2 3 4 5 … 12

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/topics/language-models/page/4/