Elamur Research⌕
☰
home / topics / Language models & pretraining / page 6

🧠 Language models & pretraining

1,079 papers · page 6 of 12 · 426–525 · the most cited first · all topics

Models that learn a language from raw text, and how they are pretrained, tuned and prompted.

1 … 5 6 7 … 12

  1. Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-Trained Language ModelsBill Yuchen Lin, Seyeon Lee, Rahul Khanna, et al.2020 · 216 citations
  2. Supervised Contrastive Learning for Pre-trained Language Model Fine-tuningBeliz Gunel, Jingfei Du, Alexis Conneau, et al.2020 · 216 citations
  3. Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language ModelsRobert Logan IV, Ivana Balazevic, Eric Wallace, et al.2022 · 215 citations
  4. Maieutic Prompting: Logically Consistent Reasoning with Recursive ExplanationsJaehun Jung, Lianhui Qin, Sean Welleck, et al.2022 · 213 citations
  5. N-gram Counts and Language Models from the Common CrawlChristian Buck, Kenneth Heafield, Bas van Ooyen2014 · 213 citations
  6. Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language ModelsMosh Levy, Alon Jacoby, Yoav Goldberg2024 · 212 citations
  7. Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, et al.2023 · 212 citations
  8. Fine-Grained Human Feedback Gives Better Rewards for Language Model TrainingZeqiu Wu, Yushi Hu, Weijia Shi, et al.2023 · 212 citations
  9. Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?Byung-Doh Oh, William Schuler2023 · 211 citations
  10. An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language ModelsNicholas Meade, Elinor Poole-Dayan, Siva Reddy2022 · 211 citations
  11. Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPOmar Khattab, Keshav Santhanam, Xiang Lisa Li, et al.2022 · 211 citations
  12. Teaching language models to support answers with verified quotesJacob Menick, Maja Trebacz, Vladimir Mikulik, et al.2022 · 211 citations
  13. Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!Yubo Ma, Yixin Cao, Yong Hong, et al.2023 · 210 citations
  14. LLM2Vec: Large Language Models Are Secretly Powerful Text EncodersParishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, et al.2024 · 209 citations
  15. Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free LunchLe Yu, Bowen Yu, Haiyang Yu, et al.2023 · 209 citations
  16. OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language ModelsAnas Awadalla, Irena Gao, Josh Gardner, et al.2023 · 209 citations
  17. Reinforced Self-Training (ReST) for Language ModelingCaglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, et al.2023 · 209 citations
  18. PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about ChangeKarthik Valmeekam, Matthew Marquez, Alberto Olmo, et al.2022 · 209 citations
  19. Compressing Context to Enhance Inference Efficiency of Large Language ModelsYucheng Li, Bo Dong, Frank Guerin, et al.2023 · 208 citations
  20. Sources of Hallucination by Large Language Models on Inference TasksNick McKenna, Tianyi Li, Liang Cheng, et al.2023 · 208 citations
  21. Active Example Selection for In-Context LearningYiming Zhang, Shi Feng, Chenhao Tan2022 · 207 citations
  22. Enabling Language Models to Fill in the BlanksChris Donahue, Mina Lee, Percy Liang2020 · 207 citations
  23. LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language ModelsYen-Ting Lin, Yun-Nung Chen2023 · 206 citations
  24. Generating Training Data with Language Models: Towards Zero-Shot Language UnderstandingYu Meng, Jiaxin Huang, Yu Zhang, et al.2022 · 206 citations
  25. Prompting GPT-3 To Be ReliableChenglei Si, Zhe Gan, Zhengyuan Yang, et al.2022 · 206 citations
  26. Challenges in Detoxifying Language ModelsJohannes Welbl, Amelia Glaese, Jonathan Uesato, et al.2021 · 206 citations
  27. RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language ModelsCheng Niu, Yuanhao Wu, Juno Zhu, et al.2024 · 204 citations
  28. LLM+P: Empowering Large Language Models with Optimal Planning ProficiencyBo Liu, Yuqian Jiang, Xiaohan Zhang, et al.2023 · 204 citations
  29. Internet-augmented language models through few-shot prompting for open-domain question answeringAngeliki Lazaridou, Elena Gribovskaya, Wojciech Stokowiec, et al.2022 · 204 citations
  30. PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document SummarizationWen Xiao, Iz Beltagy, Giuseppe Carenini, et al.2022 · 204 citations
  31. On the Effectiveness of Adapter-based Tuning for Pretrained Language Model AdaptationRuidan He, Linlin Liu, Hai Ye, et al.2021 · 204 citations
  32. Aya Dataset: An Open-Access Collection for Multilingual Instruction TuningShivalika Singh, Freddie Vargus, Daniel D'souza, et al.2024 · 202 citations
  33. How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMsYi Zeng, Hongpeng Lin, Jingwen Zhang, et al.2024 · 202 citations
  34. SALMONN: Towards Generic Hearing Abilities for Large Language ModelsChangli Tang, Wenyi Yu, Guangzhi Sun, et al.2023 · 202 citations
  35. TinyStories: How Small Can Language Models Be and Still Speak Coherent English?Ronen Eldan, Yuanzhi Li2023 · 202 citations
  36. Language Modeling for Code-Mixing: The Role of Linguistic Theory based Synthetic DataAdithya Pratapa, Gayatri Bhat, Monojit Choudhury, et al.2018 · 202 citations
  37. DeepSeek-VL: Towards Real-World Vision-Language UnderstandingHaoyu Lu, Wen Liu, Bo Zhang, et al.2024 · 201 citations
  38. Aligning Large Language Models with Human: A SurveyYufei Wang, Wanjun Zhong, Liangyou Li, et al.2023 · 199 citations
  39. Large Language Model as Attributed Training Data Generator: A Tale of Diversity and BiasYue Yu, Yuchen Zhuang, Jieyu Zhang, et al.2023 · 198 citations
  40. Model Tells You What to Discard: Adaptive KV Cache Compression for LLMsSuyu Ge, Yunan Zhang, Liyuan Liu, et al.2023 · 198 citations
  41. MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse AttentionHuiqiang Jiang, Yucheng Li, Chengruidong Zhang, et al.2024 · 197 citations
  42. Quest: Query-Aware Sparsity for Efficient Long-Context LLM InferenceJiaming Tang, Yilong Zhao, Kan Zhu, et al.2024 · 197 citations
  43. ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree SearchDan Zhang, Sining Zhoubian, Ziniu Hu, et al.2024 · 197 citations
  44. Prompt Injection attack against LLM-integrated ApplicationsYi Liu; Gelei Deng; Yuekang Li; Kailong Wang; Zihao Wang; Xiaofeng Wang; Tianwei Zhang; Yepang Liu; Haoyu Wang; Yan Zheng; Leo Yu Zhang;, Yang Liu2023 · 197 citations
  45. Scalable Extraction of Training Data from (Production) Language ModelsMilad Nasr, Nicholas Carlini, Jonathan Hayase, et al.2023 · 197 citations
  46. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?Yang Yue, Zhiqi Chen, Rui Lu, et al.2025 · 196 citations
  47. The Unreliability of Explanations in Few-shot Prompting for Textual ReasoningXi Ye, Greg Durrett2022 · 196 citations
  48. COCO-LM: Correcting and Contrasting Text Sequences for Language Model PretrainingYu Meng, Chenyan Xiong, Payal Bajaj, et al.2021 · 196 citations
  49. Structured Pruning of Large Language ModelsZiheng Wang, Jeremy Wohlwend, Tao Lei2020 · 196 citations
  50. What matters when building vision-language models?Hugo Laurençon, Leo Tronchon, Matthieu Cord, et al.2024 · 195 citations
  51. Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-TuningJesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, et al.2022 · 195 citations
  52. InstructUIE: Multi-task Instruction Tuning for Unified Information ExtractionXiao Wang, Weikang Zhou, Can Zu, et al.2023 · 194 citations
  53. Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human FeedbackViet Lai, Chien Nguyen, Nghia Ngo, et al.2023 · 194 citations
  54. RRHF: Rank Responses to Align Language Models with Human Feedback without tearsZheng Yuan, Hongyi Yuan, Chuanqi Tan, et al.2023 · 194 citations
  55. Shikra: Unleashing Multimodal LLM's Referential Dialogue MagicKeqin Chen, Zhao Zhang, Weili Zeng, et al.2023 · 194 citations
  56. Time-Aware Language Models as Temporal Knowledge BasesBhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, et al.2022 · 194 citations
  57. The emergence of number and syntax units in LSTM language modelsYair Lakretz, German Kruszewski, Theo Desbordes, et al.2019 · 194 citations
  58. Making Retrieval-Augmented Language Models Robust to Irrelevant ContextOri Yoran, Tomer Wolfson, Ori Ram, et al.2023 · 193 citations
  59. OmniQuant: Omnidirectionally Calibrated Quantization for Large Language ModelsWenqi Shao, Mengzhao Chen, Zhaoyang Zhang, et al.2023 · 193 citations
  60. LLM in a flash: Efficient Large Language Model Inference with Limited MemoryKeivan Alizadeh, Seyed Iman Mirzadeh, Dmitry Belenko, et al.2024 · 192 citations
  61. MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningJun Chen, Deyao Zhu, Xiaoqian Shen, et al.2023 · 192 citations
  62. Auto-Debias: Debiasing Masked Language Models with Automated Biased PromptsYue Guo, Yi Yang, Ahmed Abbasi2022 · 191 citations
  63. HONEST: Measuring Hurtful Sentence Completion in Language ModelsDebora Nozza, Federico Bianchi, Dirk Hovy2021 · 190 citations
  64. PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning OptimizationYidong Wang, Zhuohao Yu, Zhengran Zeng, et al.2023 · 189 citations
  65. When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language ModelsBenjamin Muller, Antonios Anastasopoulos, Benoît Sagot, et al.2021 · 189 citations
  66. Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense DisambiguationDaniel Loureiro, Alípio Jorge2019 · 189 citations
  67. Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMsXingyu Chen, Jiahao Xu, Tian Liang, et al.2024 · 188 citations
  68. LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation MethodsHaitao Li, Qian Dong, Junjie Chen, et al.2024 · 188 citations
  69. Refusal in Language Models Is Mediated by a Single DirectionAndy Arditi, Oscar Obeso, Aaquib Syed, et al.2024 · 188 citations
  70. SGPT: GPT Sentence Embeddings for Semantic SearchNiklas Muennighoff2022 · 187 citations
  71. Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language ModelsMatt Deitke, Christopher Clark, Sangho Lee, et al.2024 · 184 citations
  72. Multilingual Jailbreak Challenges in Large Language ModelsYue Deng, Wenxuan Zhang, Sinno Jialin Pan, et al.2023 · 183 citations
  73. Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language ModelsKushal Tirumala, Aram H. Markosyan, Luke Zettlemoyer, et al.2022 · 183 citations
  74. Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP ResearchersChenglei Si, Diyi Yang, Tatsunori Hashimoto2024 · 182 citations
  75. Large Language Models are few(1)-shot Table ReasonersWenhu Chen2023 · 182 citations
  76. Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language ModelsYunfei Chu, Jin Xu, Xiaohuan Zhou, et al.2023 · 182 citations
  77. Pre-training via ParaphrasingMike Lewis, Marjan Ghazvininejad, Gargi Ghosh, et al.2020 · 181 citations
  78. GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language ModelsArchiki Prasad, Peter Hase, Xiang Zhou, et al.2023 · 180 citations
  79. Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction TuningFuxiao Liu, Kevin Lin, Linjie Li, et al.2023 · 180 citations
  80. Knowledge Conflicts for LLMs: A SurveyRongwu Xu, Zehan Qi, Zhijiang Guo, et al.2024 · 179 citations
  81. Adapting Language Models to Compress ContextsAlexis Chevalier, Alexander Wettig, Anirudh Ajith, et al.2023 · 179 citations
  82. CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 LanguagesThuat Nguyen, Chien Van Nguyen, Viet Dac Lai, et al.2023 · 179 citations
  83. The False Promise of Imitating Proprietary LLMsArnav Gudibande, Eric Wallace, Charlie Snell, et al.2023 · 178 citations
  84. Investigating Cultural Alignment of Large Language ModelsBadr AlKhamissi, Muhammad ElNokrashy, Mai Alkhamissi, et al.2024 · 177 citations
  85. Two Tales of Persona in LLMs: A Survey of Role-Playing and PersonalizationYu-Min Tseng, Yu-Chao Huang, Teng-Yun Hsiao, et al.2024 · 177 citations
  86. Effective Long-Context Scaling of Foundation ModelsWenhan Xiong, Jingyu Liu, Igor Molybog, et al.2023 · 177 citations
  87. Evaluating Large Language Models: A Comprehensive SurveyZishan Guo, Renren Jin, Chuang Liu, et al.2023 · 177 citations
  88. ToolQA: A Dataset for LLM Question Answering with External ToolsYuchen Zhuang, Yue Yu, Kuan Wang, et al.2023 · 177 citations
  89. Pre-training Multilingual Neural Machine Translation by Leveraging Alignment InformationZehui Lin, Xiao Pan, Mingxuan Wang, et al.2020 · 177 citations
  90. Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language ModelsNora Kassner, Philipp Dufter, Hinrich Schütze2021 · 175 citations
  91. Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language ModelsQiguang Chen, Libo Qin, Jinhao Liu, et al.2025 · 174 citations
  92. Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic LanguagesSumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, et al.2023 · 174 citations
  93. What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction TuningWei Liu, Weihao Zeng, Keqing He, et al.2023 · 174 citations
  94. UNKs Everywhere: Adapting Multilingual Language Models to New ScriptsJonas Pfeiffer, Ivan Vulić, Iryna Gurevych, et al.2021 · 174 citations
  95. Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMsShengbang Tong, Ellis Brown, Penghao Wu, et al.2024 · 173 citations
  96. Language Models are Few-shot Multilingual LearnersGenta Indra Winata, Andrea Madotto, Zhaojiang Lin, et al.2021 · 173 citations
  97. ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated CasesQiaoyu Tang, Ziliang Deng, Hongyu Lin, et al.2023 · 172 citations
  98. Lifting the Curse of Multilinguality by Pre-training Modular TransformersJonas Pfeiffer, Naman Goyal, Xi Victoria Lin, et al.2022 · 172 citations
  99. SmolLM2: When Smol Goes Big — Data-Centric Training of a Small Language ModelLoubna Ben Allal, Anton Lozhkov, Elie Bakouch, et al.2025 · 171 citations
  100. MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn DialoguesGe Bai, Jie Liu, Xingyuan Bu, et al.2024 · 171 citations

1 … 5 6 7 … 12

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/topics/language-models/page/6/