🧠 Language models & pretraining

1,079 papers · page 8 of 12 · 626–725 · the most cited first · all topics

Models that learn a language from raw text, and how they are pretrained, tuned and prompted.

← prev 1 … 7 8 9 … 12 next →

626. LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation TasksAnna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, et al.2024 · 137 citations
627. Efficient Training of Language Models to Fill in the MiddleMohammad Bavarian, Heewoo Jun, Nikolas Tezak, et al.2022 · 137 citations
628. A fine-grained comparison of pragmatic language understanding in humans and language modelsJennifer Hu, Sammy Floyd, Olessia Jouravlev, et al.2023 · 136 citations
629. Detecting Language Model Attacks with PerplexityGabriel Alon, Michael Kamfonas2023 · 136 citations
630. MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language FeedbackXingyao Wang, Zihan Wang, Jiateng Liu, et al.2023 · 136 citations
631. mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality CollaborationQinghao Ye, Haiyang Xu, Jiabo Ye, et al.2023 · 136 citations
632. Attributed Question Answering: Evaluation and Modeling for Attributed Large Language ModelsBernd Bohnet, Vinh Q. Tran, Pat Verga, et al.2022 · 136 citations
633. Byte Pair Encoding is Suboptimal for Language Model PretrainingKaj Bostrom, Greg Durrett2020 · 136 citations
634. DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal UnderstandingZhiyu Wu, Xiaokang Chen, Zizheng Pan, et al.2024 · 135 citations
635. RAFT: Adapting Language Model to Domain Specific RAGTianjun Zhang, Shishir G. Patil, Naman Jain, et al.2024 · 135 citations
636. Token-Budget-Aware LLM ReasoningTingxu Han, Zhenting Wang, Chunrong Fang, et al.2024 · 135 citations
637. SafetyBench: Evaluating the Safety of Large Language ModelsZhexin Zhang, Leqi Lei, Lindong Wu, et al.2023 · 135 citations
638. Many-Shot In-Context LearningRishabh Agarwal, Avi Singh, Lei M. Zhang, et al.2024 · 134 citations
639. Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language ModelsPaul Röttger, Valentin Hofmann, Valentina Pyatkin, et al.2024 · 134 citations
640. Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse ModelsPat Verga, Sebastian Hofstatter, Sophia Althammer, et al.2024 · 134 citations
641. Simple and Effective Masked Diffusion Language ModelsSubham Sekhar Sahoo, Marianne Arriola, Yair Schiff, et al.2024 · 134 citations
642. Beyond Human Data: Scaling Self-Training for Problem-Solving with Language ModelsAvi Singh, John D. Co-Reyes, Rishabh Agarwal, et al.2023 · 134 citations
643. InFoBench: Evaluating Instruction Following Ability in Large Language ModelsYiwei Qin, Kaiqiang Song, Yebowen Hu, et al.2024 · 133 citations
644. TOFU: A Task of Fictitious Unlearning for LLMsPratyush Maini, Zhili Feng, Avi Schwarzschild, et al.2024 · 133 citations
645. Prompting is not a substitute for probability measurements in large language modelsJennifer Hu, Roger Levy2023 · 133 citations
646. Long-context LLMs Struggle with Long In-context LearningTianle Li, Ge Zhang, Quy Duc Do, et al.2024 · 132 citations
647. EAGLE-2: Faster Inference of Language Models with Dynamic Draft TreesYuhui Li, Fangyun Wei, Chao Zhang, et al.2024 · 131 citations
648. EuroLLM: Multilingual Language Models for EuropePedro Henrique Martins, Patrick Fernandes, Joao Alves, et al.2024 · 131 citations
649. Understanding the Effects of RLHF on LLM Generalisation and DiversityRobert Kirk, Ishita Mediratta, Christoforos Nalmpantis, et al.2023 · 131 citations
650. Extracting Latent Steering Vectors from Pretrained Language ModelsNishant Subramani, Nivedita Suresh, Matthew Peters2022 · 131 citations
651. FreshLLMs: Refreshing Large Language Models with Search Engine AugmentationTu Vu, Mohit Iyyer, Xuezhi Wang, et al.2023 · 130 citations
652. Secrets of RLHF in Large Language Models Part I: PPORui Zheng, Shihan Dou, Songyang Gao, et al.2023 · 130 citations
653. Improving Language Model Negotiation with Self-Play and In-Context Learning from AI FeedbackYao Fu, Hao Peng, Tushar Khot, et al.2023 · 129 citations
654. Retrieval meets Long Context Large Language ModelsPeng Xu, Wei Ping, Xianchao Wu, et al.2023 · 129 citations
655. AceGPT, Localizing Large Language Models in ArabicHuang Huang, Fei Yu, Jianqing Zhu, et al.2024 · 127 citations
656. Time Travel in LLMs: Tracing Data Contamination in Large Language ModelsShahriar Golchin, Mihai Surdeanu2023 · 127 citations
657. Knowledge Unlearning for Mitigating Privacy Risks in Language ModelsJoel Jang, Dongkeun Yoon, Sohee Yang, et al.2022 · 126 citations
658. InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context MemoryChaojun Xiao, Pengle Zhang, Xu Han, et al.2024 · 125 citations
659. Jailbreak and Guard Aligned Language Models with Only Few In-Context DemonstrationsZeming Wei, Yifei Wang, Ang Li, et al.2023 · 125 citations
660. Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow InstructionsFederico Bianchi, Mirac Suzgun, Giuseppe Attanasio, et al.2023 · 124 citations
661. Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language ModelsLei Li, Yuqi Wang, Runxin Xu, et al.2024 · 123 citations
662. Large Language Model Alignment: A SurveyTianhao Shen, Renren Jin, Yufei Huang, et al.2023 · 123 citations
663. Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across LanguagesLibo Qin, Qiguang Chen, Fuxuan Wei, et al.2023 · 122 citations
664. Automated evaluation of written discourse coherence using GPT-4Ben Naismith, Phoebe Mulcaire, Jill Burstein2023 · 120 citations
665. Fine-tuning Language Models for FactualityKatherine Tian, Eric Mitchell, Huaxiu Yao, et al.2023 · 120 citations
666. The Impact of Reasoning Step Length on Large Language ModelsMingyu Jin, Qinkai Yu, Dong Shu, et al.2024 · 119 citations
667. Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question AnsweringJinheon Baek, Alham Fikri Aji, Amir Saffari2023 · 118 citations
668. Chameleon: Mixed-Modal Early-Fusion Foundation ModelsChameleon Team2024 · 117 citations
669. Pretraining Language Models with Human PreferencesTomasz Korbak, Kejian Shi, Angelica Chen, et al.2023 · 117 citations
670. QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice CodebooksAlbert Tseng, Jerry Chee, Qingyao Sun, et al.2024 · 116 citations
671. HealthBench: Evaluating Large Language Models Towards Improved Human HealthRahul K. Arora, Jason Wei, Rebecca Soskin Hicks, et al.2025 · 115 citations
672. Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language CapabilitiesKazuki Fujii, Taishi Nakamura, Mengsay Loem, et al.2024 · 114 citations
673. Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language ModelsYangzhen Wu, Zhiqing Sun, Shanda Li, et al.2024 · 114 citations
674. SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language ModelsLijun Li, Bowen Dong, Ruohui Wang, et al.2024 · 114 citations
675. Demystifying Long Chain-of-Thought Reasoning in LLMsEdward Yeo, Yuxuan Tong, Morry Niu, et al.2025 · 113 citations
676. ALLaM: Large Language Models for Arabic and EnglishM Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, et al.2024 · 113 citations
677. Quiet-STaR: Language models can teach themselves to think before speaking2024 · 113 citations
678. Break the Sequential Dependency of LLM Inference Using Lookahead DecodingYichao Fu, Peter Bailis, Ion Stoica, et al.2024 · 112 citations
679. DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic DataHuajian Xin, Daya Guo, Zhihong Shao, et al.2024 · 111 citations
680. DistiLLM: Towards Streamlined Distillation for Large Language ModelsJongwoo Ko, Sungnyun Kim, Tianyi Chen, et al.2024 · 111 citations
681. Certifying LLM Safety against Adversarial PromptingAounon Kumar, Chirag Agarwal, Suraj Srinivas, et al.2023 · 110 citations
682. Olmo 3Team Olmo: Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, et al.2025 · 109 citations
683. DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming HeadsGuangxuan Xiao, Jiaming Tang, Jingwei Zuo, et al.2024 · 109 citations
684. Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-VerificationAojun Zhou, Ke Wang, Zimu Lu, et al.2023 · 109 citations
685. RestGPT: Connecting Large Language Models with Real-World RESTful APIsYifan Song, Weimin Xiong, Dawei Zhu, et al.2023 · 108 citations
686. Data Engineering for Scaling Language Models to 128K ContextYao Fu, Rameswar Panda, Xinyao Niu, et al.2024 · 107 citations
687. GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem SolversQintong Li, Leyang Cui, Xueliang Zhao, et al.2024 · 107 citations
688. Evaluating the Performance of Large Language Models on GAOKAO BenchmarkXiaotian Zhang, Chunyang Li, Yi Zong, et al.2023 · 107 citations
689. Physics of Language Models: Part 3.1, Knowledge Storage and ExtractionZeyuan Allen-Zhu, Yuanzhi Li2023 · 107 citations
690. French CrowS-Pairs: Extending a challenge dataset for measuring social bias in masked language models to a language other than EnglishAurélie Névéol, Yoann Dupont, Julien Bezançon, et al.2022 · 107 citations
691. GPT detectors are biased against non-native English writersWeixin Liang, Mert Yuksekgonul, Yining Mao, et al.2023 · 106 citations
692. TokenSkip: Controllable Chain-of-Thought Compression in LLMsHeming Xia, Chak Tou Leong, Wenjie Wang, et al.2025 · 105 citations
693. Skywork-Reward: Bag of Tricks for Reward Modeling in LLMsChris Yuhao Liu, Liang Zeng, Jiacai Liu, et al.2024 · 104 citations
694. Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement LearningTian Xie, Zitian Gao, Qingnan Ren, et al.2025 · 103 citations
695. Tokenizer Choice For LLM Training: Negligible or Crucial?Mehdi Ali, Michael Fromm, Klaudia Thellmann, et al.2024 · 102 citations
696. Scaling Laws for TransferDanny Hernandez, Jared Kaplan, Tom Henighan, et al.2021 · 102 citations
697. Rethinking Benchmark and Contamination for Language Models with Rephrased SamplesShuo Yang, Wei-Lin Chiang, Lianmin Zheng, et al.2023 · 100 citations
698. MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical ReasoningKe Wang, Houxing Ren, Aojun Zhou, et al.2023 · 98 citations
699. LongAlign: A Recipe for Long Context Alignment of Large Language ModelsYushi Bai, Xin Lv, Jiajie Zhang, et al.2024 · 97 citations
700. Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMsXin Lai, Zhuotao Tian, Yukang Chen, et al.2024 · 97 citations
701. Skywork: A More Open Bilingual Foundation ModelTianwen Wei, Liang Zhao, Lichang Zhang, et al.2023 · 91 citations
702. WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language modelsBenjamin Minixhofer, Fabian Paischer, Navid Rekabsaz2022 · 90 citations
703. Language Modeling Is CompressionGregoire Deletang, Anian Ruoss, Paul-Ambroise Duquenne, et al.2023 · 89 citations
704. AStitchInLanguageModels: Dataset and Methods for the Exploration of Idiomaticity in Pre-Trained Language ModelsHarish Tayyar Madabushi, Edward Gow-Smith, Carolina Scarton, et al.2021 · 86 citations
705. Language Modeling for Morphologically Rich Languages: Character-Aware Modeling for Word-Level PredictionDaniela Gerz, Ivan Vulić, Edoardo Ponti, et al.2018 · 82 citations
706. D4: Improving LLM Pretraining via Document De-Duplication and DiversificationKushal Tirumala, Daniel Simig, Armen Aghajanyan, et al.2023 · 79 citations
707. Are Pretrained Language Models Symbolic Reasoners over Knowledge?Nora Kassner, Benno Krojer, Hinrich Schütze2020 · 79 citations
708. PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics CapabilitiesSettaluri Sravanthi, Meet Doshi, Pavan Tankala, et al.2024 · 75 citations
709. Dipping PLMs Sauce: Bridging Structure and Text for Effective Knowledge Graph Completion via Conditional Soft PromptingChen Chen, Yufei Wang, Aixin Sun, et al.2023 · 75 citations
710. MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language ModelsWai-Chung Kwan, Xingshan Zeng, Yuxin Jiang, et al.2024 · 66 citations
711. AlephBERT: Language Model Pre-training and Evaluation from Sub-Word to Sentence LevelAmit Seker, Elron Bandel, Dan Bareket, et al.2022 · 57 citations
712. AlephBERT: A Hebrew Large Pre-Trained Language Model to Start-off your Hebrew NLP Application WithAmit Seker, Elron Bandel, Dan Bareket, et al.2021 · 56 citations
713. Tree Linearization in English: Improving Language Model Based ApproachesKatja Filippova, Michael Strube2009 · 56 citations
714. Tokenization counts: the impact of tokenization on arithmetic in frontier LLMsAaditya K. Singh, DJ Strouse2024 · 53 citations
715. The Unreasonable Effectiveness of Transformer Language Models in Grammatical Error CorrectionDimitris Alikaniotis, Vipul Raheja2019 · 52 citations
716. SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language ModelsXin Zhang, Dong Zhang, Shimin Li, et al.2023 · 51 citations
717. Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and ChallengesBolei Ma, Yuting Li, Wei Zhou, et al.2025 · 49 citations
718. Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling LawsNikhil Sardana, Jacob Portes, Sasha Doubov, et al.2024 · 47 citations
719. Metaphor Understanding Challenge Dataset for LLMsXiaoyu Tong, Rochelle Choenni, Martha Lewis, et al.2024 · 47 citations
720. Counting the Bugs in ChatGPT's Wugs: A Multilingual Investigation into the Morphological Capabilities of a Large Language ModelLeonie Weissweiler, Valentin Hofmann, Anjali Kantharuban2023 · 47 citations
721. Bridging the Preference Gap between Retrievers and LLMsZixuan Ke, Weize Kong, Cheng Li, et al.2024 · 45 citations
722. Getting the most out of your tokenizer for pre-training and domain adaptationGautier Dagan, Gabriel Synnaeve, Baptiste Roziere2024 · 44 citations
723. Pragmatic competence of pre-trained language models through the lens of discourse connectivesLalchand Pandia, Yan Cong, Allyson Ettinger2021 · 42 citations
724. Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time ScalingRunze Liu, Junqi Gao, Jian Zhao, et al.2025 · 41 citations
725. Scaling Laws for Generative Mixed-Modal Language ModelsArmen Aghajanyan, Lili Yu, Alexis Conneau, et al.2023 · 39 citations

← prev 1 … 7 8 9 … 12 next →

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/topics/language-models/page/8/