Elamur Research⌕
☰
home / topics / Language models & pretraining / page 5

🧠 Language models & pretraining

1,079 papers · page 5 of 12 · 326–425 · the most cited first · all topics

Models that learn a language from raw text, and how they are pretrained, tuned and prompted.

1 … 4 5 6 … 12

  1. Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for LittleKoustuv Sinha, Robin Jia, Dieuwke Hupkes, et al.2021 · 292 citations
  2. SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized OptimizationHaoming Jiang, Pengcheng He, Weizhu Chen, et al.2020 · 291 citations
  3. Neural language models as psycholinguistic subjects: Representations of syntactic stateRichard Futrell, Ethan Wilcox, Takashi Morita, et al.2019 · 291 citations
  4. MAmmoTH: Building Math Generalist Models through Hybrid Instruction TuningXiang Yue, Xingwei Qu, Ge Zhang, et al.2023 · 288 citations
  5. On-Policy Distillation of Language Models: Learning from Self-Generated MistakesRishabh Agarwal, Nino Vieillard, Yongchao Zhou, et al.2023 · 288 citations
  6. Can language models learn from explanations in context?Andrew Lampinen, Ishita Dasgupta, Stephanie Chan, et al.2022 · 288 citations
  7. API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMsMinghao Li, Yingxiu Zhao, Bowen Yu, et al.2023 · 281 citations
  8. Generate rather than Retrieve: Large Language Models are Strong Context GeneratorsWenhao Yu, Dan Iter, Shuohang Wang, et al.2022 · 280 citations
  9. Large Language Models for Mathematical Reasoning: Progresses and ChallengesJanice Ahn, Rishu Verma, Renze Lou, et al.2024 · 279 citations
  10. HuatuoGPT, Towards Taming Language Model to Be a DoctorHongbo Zhang, Junying Chen, Feng Jiang, et al.2023 · 279 citations
  11. Otter: A Multi-Modal Model with In-Context Instruction TuningBo Li, Yuanhan Zhang, Liangyu Chen, et al.2023 · 279 citations
  12. Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?Yada Pruksachatkun, Jason Phang, Haokun Liu, et al.2020 · 279 citations
  13. Larger language models do in-context learning differentlyJerry Wei, Jason Wei, Yi Tay, et al.2023 · 277 citations
  14. An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language ModelsLiang Chen, Haozhe Zhao, Tianyu Liu, et al.2024 · 276 citations
  15. What do RNN Language Models Learn about Filler-Gap Dependencies?Ethan Wilcox, Roger Levy, Takashi Morita, et al.2018 · 276 citations
  16. LLM-QAT: Data-Free Quantization Aware Training for Large Language ModelsZechun Liu, Barlas Oguz, Changsheng Zhao, et al.2024 · 275 citations
  17. Llemma: An Open Language Model For MathematicsZhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, et al.2023 · 275 citations
  18. Text Classification Using Label Names Only: A Language Model Self-Training ApproachYu Meng, Yunyi Zhang, Jiaxin Huang, et al.2020 · 274 citations
  19. JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language ModelsPatrick Chao, Edoardo Debenedetti, Alexander Robey, et al.2024 · 273 citations
  20. Kosmos-2: Grounding Multimodal Large Language Models to the WorldZhiliang Peng, Wenhui Wang, Li Dong, et al.2023 · 272 citations
  21. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding HeadsTianle Cai, Yuhong Li, Zhengyang Geng, et al.2024 · 271 citations
  22. XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and BeyondFrancesco Barbieri, Luis Espinosa Anke, Jose Camacho-Collados2022 · 271 citations
  23. InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-TrainingZewen Chi, Li Dong, Furu Wei, et al.2020 · 269 citations
  24. Having Beer after Prayer? Measuring Cultural Bias in Large Language ModelsTarek Naous, Michael J. Ryan, Alan Ritter, et al.2023 · 268 citations
  25. An Explanation of In-context Learning as Implicit Bayesian InferenceSang Michael Xie, Aditi Raghunathan, Percy Liang, et al.2021 · 268 citations
  26. Injecting Numerical Reasoning Skills into Language ModelsMor Geva, Ankit Gupta, Jonathan Berant2020 · 268 citations
  27. MiniCPM-V: A GPT-4V Level MLLM on Your PhoneYuan Yao, Tianyu Yu, Ao Zhang, et al.2024 · 267 citations
  28. Prometheus: Inducing Fine-grained Evaluation Capability in Language ModelsSeungone Kim, Jamin Shin, Yejin Cho, et al.2023 · 267 citations
  29. Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What MattersBoshi Wang, Sewon Min, Xiang Deng, et al.2023 · 267 citations
  30. GPT3Mix: Leveraging Large-scale Language Models for Text AugmentationKang Min Yoo, Dongju Park, Jaewook Kang, et al.2021 · 267 citations
  31. LongLoRA: Efficient Fine-tuning of Long-Context Large Language ModelsYukang Chen, Shengju Qian, Haotian Tang, et al.2023 · 265 citations
  32. Small Data? No Problem! Exploring the Viability of Pretrained Multilingual Language Models for Low-resourced LanguagesKelechi Ogueji, Yuxin Zhu, Jimmy Lin2021 · 265 citations
  33. HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language ModelsBernal Jimenez Gutierrez, Yiheng Shu, Yu Gu, et al.2024 · 264 citations
  34. Are We on the Right Way for Evaluating Large Vision-Language Models?Lin Chen, Jinsong Li, Xiaoyi Dong, et al.2024 · 263 citations
  35. Towards Understanding and Mitigating Social Biases in Language ModelsPaul Pu Liang, Chiyu Wu, Louis-Philippe Morency, et al.2021 · 263 citations
  36. GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language ModelsIman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, et al.2024 · 262 citations
  37. Towards Measuring the Representation of Subjective Global Opinions in Language ModelsEsin Durmus, Karina Nguyen, Thomas I. Liao, et al.2023 · 262 citations
  38. DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language ModelsYung-Sung Chuang, Yujia Xie, Hongyin Luo, et al.2023 · 261 citations
  39. DSPy: Compiling Declarative Language Model Calls into Self-Improving PipelinesOmar Khattab, Arnav Singhvi, Paridhi Maheshwari, et al.2023 · 261 citations
  40. Condenser: a Pre-training Architecture for Dense RetrievalLuyu Gao, Jamie Callan2021 · 261 citations
  41. Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual TasksHaoyang Huang, Yaobo Liang, Nan Duan, et al.2019 · 261 citations
  42. Large Language Models for Data Annotation and Synthesis: A SurveyZhen Tan, Dawei Li, Song Wang, et al.2024 · 259 citations
  43. On Extractive and Abstractive Neural Document Summarization with Transformer Language ModelsJonathan Pilault, Raymond Li, Sandeep Subramanian, et al.2020 · 259 citations
  44. MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training StrategiesShengding Hu, Yuge Tu, Xu Han, et al.2024 · 258 citations
  45. Large Language Models Sensitivity to The Order of Options in Multiple-Choice QuestionsPouya Pezeshkpour, Estevam Hruschka2023 · 256 citations
  46. TAPEX: Table Pre-training via Learning a Neural SQL ExecutorQian Liu, Bei Chen, Jiaqi Guo, et al.2021 · 254 citations
  47. QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMsSaleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, et al.2024 · 252 citations
  48. GPT-RE: In-context Learning for Relation Extraction using Large Language ModelsZhen Wan, Fei Cheng, Zhuoyuan Mao, et al.2023 · 252 citations
  49. Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-OptimizersDamai Dai, Yutao Sun, Li Dong, et al.2023 · 252 citations
  50. Personality Traits in Large Language ModelsGreg Serapio-Garcia, Mustafa Safdari, Clement Crepy, et al.2023 · 251 citations
  51. Generating Datasets with Pretrained Language ModelsTimo Schick, Hinrich Schutze2021 · 250 citations
  52. BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language ModelAlex Wang, Kyunghyun Cho2019 · 249 citations
  53. LeanDojo: Theorem Proving with Retrieval-Augmented Language ModelsKaiyu Yang, Aidan M. Swope, Alex Gu, et al.2023 · 248 citations
  54. OpenAI GPT-5 System CardAaditya Singh, Adam Fry, Adam Perelman, et al.2025 · 247 citations
  55. RedPajama: an Open Dataset for Training Large Language ModelsMaurice Weber, Daniel Fu, Quentin Anthony, et al.2024 · 247 citations
  56. A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language ModelsS.M Towhidul Islam Tonmoy, S M Mehedi Zaman, Vinija Jain, et al.2024 · 246 citations
  57. DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT ModelsBoxin Wang, Weixin Chen, Hengzhi Pei, et al.2023 · 246 citations
  58. Aya Model: An Instruction Finetuned Open-Access Multilingual Language ModelAhmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, et al.2024 · 245 citations
  59. NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each BenchmarkOscar Sainz, Jon Campos, Iker García-Ferrero, et al.2023 · 245 citations
  60. Tree of Attacks: Jailbreaking Black-Box LLMs AutomaticallyAnay Mehrotra, Manolis Zampetakis, Paul Kassianik, et al.2023 · 245 citations
  61. Ignore Previous Prompt: Attack Techniques For Language ModelsFabio Perez, Ian Ribeiro2022 · 244 citations
  62. Factuality Challenges in the Era of Large Language ModelsIsabelle Augenstein, Timothy Baldwin, Meeyoung Cha, et al.2023 · 241 citations
  63. LaMP: When Large Language Models Meet PersonalizationAlireza Salemi, Sheshera Mysore, Michael Bendersky, et al.2024 · 239 citations
  64. Language Models can Solve Computer TasksGeunwoo Kim, Pierre Baldi, Stephen McAleer2023 · 239 citations
  65. RAFT: Reward rAnked FineTuning for Generative Foundation Model AlignmentHanze Dong, Wei Xiong, Deepanshu Goyal, et al.2023 · 239 citations
  66. Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-ThoughtAbulhair Saparov, He He2022 · 238 citations
  67. Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-trainingOshin Agarwal, Heming Ge, Siamak Shakeri, et al.2021 · 238 citations
  68. Multimodal Few-Shot Learning with Frozen Language ModelsMaria Tsimpoukelli, Jacob Menick, Serkan Cabi, et al.2021 · 238 citations
  69. Noisy Channel Language Model Prompting for Few-Shot Text ClassificationSewon Min, Mike Lewis, Hannaneh Hajishirzi, et al.2021 · 238 citations
  70. OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of GeneralizationSrinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, et al.2022 · 237 citations
  71. On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A SurveyLin Long, Rui Wang, Ruixuan Xiao, et al.2024 · 236 citations
  72. Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language ModelsMyra Cheng, Esin Durmus, Dan Jurafsky2023 · 236 citations
  73. LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation DatasetLianmin Zheng, Wei-Lin Chiang, Ying Sheng, et al.2023 · 235 citations
  74. Generative Language Modeling for Automated Theorem ProvingStanislas Polu, Ilya Sutskever2020 · 235 citations
  75. Evaluating Open-Domain Question Answering in the Era of Large Language ModelsEhsan Kamalloo, Nouha Dziri, Charles Clarke, et al.2023 · 233 citations
  76. Qwen2-Audio Technical ReportYunfei Chu, Jin Xu, Qian Yang, et al.2024 · 232 citations
  77. Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human SupervisionZhiqing Sun, Yikang Shen, Qinhong Zhou, et al.2023 · 232 citations
  78. MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?Renrui Zhang, Dongzhi Jiang, Yichi Zhang, et al.2024 · 231 citations
  79. How Context Affects Language Models' Factual PredictionsFabio Petroni, Patrick Lewis, Aleksandra Piktus, et al.2020 · 231 citations
  80. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving PerformanceLingjiao Chen, Matei Zaharia, James Zou2023 · 229 citations
  81. XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language ModelsPaul Rottger, Hannah Rose Kirk, Bertie Vidgen, et al.2023 · 227 citations
  82. TrustLLM: Trustworthiness in Large Language ModelsYue Huang, Lichao Sun, Haoran Wang, et al.2024 · 226 citations
  83. Prompt for Extraction? PAIE: Prompting Argument Interaction for Event Argument ExtractionYubo Ma, Zehao Wang, Yixin Cao, et al.2022 · 225 citations
  84. Baseline Defenses for Adversarial Attacks Against Aligned Language ModelsNeel Jain, Avi Schwarzschild, Yuxin Wen, et al.2023 · 224 citations
  85. What learning algorithm is in-context learning? Investigations with linear modelsEkin Akyürek, Dale Schuurmans, Jacob Andreas, et al.2022 · 224 citations
  86. Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought PromptingHaoyang Huang, Tianyi Tang, Dongdong Zhang, et al.2023 · 223 citations
  87. Improving Text Simplification Language Modeling Using Unsimplified Text DataDavid Kauchak2013 · 223 citations
  88. Specializing Smaller Language Models towards Multi-Step ReasoningYao Fu, Hao Peng, Litu Ou, et al.2023 · 222 citations
  89. A Survey of Hallucination in Large Foundation ModelsVipula Rawte, Amit Sheth, Amitava Das2023 · 221 citations
  90. How Language Model Hallucinations Can SnowballMuru Zhang, Ofir Press, William Merrill, et al.2023 · 221 citations
  91. Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language ModelsSeungone Kim, Juyoung Suk, Shayne Longpre, et al.2024 · 220 citations
  92. Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt InjectionKai Greshake, Sahar Abdelnabi, Shailesh Mishra, et al.2023 · 220 citations
  93. Steering Language Models With Activation EngineeringAlexander Matt Turner, Lisa Thiergart, Gavin Leech, et al.2023 · 220 citations
  94. NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding ModelsChankyu Lee, Rajarshi Roy, Mengyao Xu, et al.2024 · 219 citations
  95. Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' AlignmentYang Liu; Yuanshun Yao; Jean-Francois Ton; Xiaoying Zhang; Ruocheng Guo; Hao Cheng; Yegor Klochkov; Muhammad Faaiz Taufiq;, Hang Li2023 · 219 citations
  96. Multilingual Translation from Denoising Pre-TrainingYuqing Tang, Chau Tran, Xian Li, et al.2021 · 219 citations
  97. Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language ModelsNeha Sengupta, Sunil Kumar Sahu, Bokang Jia, et al.2023 · 217 citations
  98. Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video UnderstandingHang Zhang, Xin Li, Lidong Bing2023 · 217 citations
  99. Distilling Reasoning Capabilities into Smaller Language ModelsKumar Shridhar, Alessandro Stolfo, Mrinmaya Sachan2023 · 216 citations
  100. Making Language Models Better Reasoners with Step-Aware VerifierYifei Li, Zeqi Lin, Shizhuo Zhang, et al.2023 · 216 citations

1 … 4 5 6 … 12

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/topics/language-models/page/5/