Elamur Research⌕
☰
home / papers / DistilBERT, a distilled version of BERT: smaller… / cited by

Who cites «DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter»

2,404 papers of our library cite it (of the 149,439 whose bibliographies we read) · 284 with a page here, the most cited first · page 1 of 3 · 1–100 · ← back to «DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter»

1 2 3

  1. Transformers: State-of-the-Art Natural Language ProcessingConference on Empirical Methods in Natural Language Processing (EMNLP)Thomas Wolf, Lysandre Debut, Victor Sanh, et al.2020 · 15,755 citations
  2. Language Models are Few-Shot LearnersarXiv:2005.14165Tom B. Brown, Benjamin Mann, Nick Ryder, et al.2020 · 14,178 citations
  3. Exploring the Limits of Transfer Learning with a Unified Text-to-Text TransformerarXiv:1910.10683Colin Raffel, Noam Shazeer, Adam Roberts, et al.2019 · 11,325 citations
  4. The Llama 3 Herd of ModelsarXiv:2407.21783Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al.2024 · 7,682 citations
  5. On the Opportunities and Risks of Foundation ModelsarXiv:2108.07258Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, et al.2021 · 3,160 citations
  6. TinyBERT: Distilling BERT for Natural Language UnderstandingFindings of the Association for Computational LinguisticsXiaoqi Jiao, Yichun Yin, Lifeng Shang, et al.2020 · 2,378 citations
  7. COMET: A Neural Framework for MT EvaluationRicardo Rei, Craig Stewart, Ana C Farinha, et al.2020 · 1,792 citations
  8. Pre-trained Models for Natural Language Processing: A SurveyarXiv:2003.08271 · SCIENCE CHINA Technological Sciences,2020, 63, 1872-1897Xipeng Qiu, Tianxiang Sun, Yige Xu, et al.2020 · 1,759 citations
  9. Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A SurveyACM Computing SurveysBonan Min, Hayley Ross, Elior Sulem, et al.2021 · 1,556 citations
  10. Efficient Transformers: A SurveyACM Computing SurveysYi Tay, Mostafa Dehghani, Dara Bahri, et al.2020 · 1,539 citations
  11. It's Not Just Size That Matters: Small Language Models Are Also Few-Shot LearnersConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Timo Schick, Hinrich Schütze2021 · 1,132 citations
  12. Quantifying Attention Flow in TransformersAnnual Meeting of the Association for Computational Linguistics (ACL)Samira Abnar, Willem Zuidema2020 · 1,064 citations
  13. MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited DevicesAnnual Meeting of the Association for Computational Linguistics (ACL)Zhiqing Sun, Hongkun Yu, Xiaodan Song, et al.2020 · 1,016 citations
  14. Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and BeyondACM Transactions on Knowledge Discovery from DataJingfeng Yang, Hongye Jin, Ruixiang Tang, et al.2023 · 814 citations
  15. GLM-130B: An Open Bilingual Pre-trained ModelarXiv:2210.02414Aohan Zeng, Xiao Liu, Zhengxiao Du, et al.2022 · 786 citations
  16. Efficient Intent Detection with Dual Sentence EncodersWorkshop on Natural Language Processing for Conversational AIIñigo Casanueva, Tadas Temčinas, Daniela Gerz, et al.2020 · 628 citations
  17. FNet: Mixing Tokens with Fourier TransformsConference of the North American Chapter of the Association for Computational Linguistics (NAACL)James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, et al.2022 · 597 citations
  18. DeCLUTR: Deep Contrastive Learning for Unsupervised Textual RepresentationsAnnual Meeting of the Association for Computational Linguistics (ACL)John Giorgi, Osvald Nitski, Bo Wang, et al.2021 · 516 citations
  19. DeeBERT: Dynamic Early Exiting for Accelerating BERT InferenceAnnual Meeting of the Association for Computational Linguistics (ACL)Ji Xin, Raphael Tang, Jaejun Lee, et al.2020 · 487 citations
  20. Pretrained Transformers Improve Out-of-Distribution RobustnessAnnual Meeting of the Association for Computational Linguistics (ACL)Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, et al.2020 · 452 citations
  21. Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware SamplingInternational ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)Sebastian Hofstatter, Sheng-Chieh Lin, Jheng-Hong Yang, et al.2021 · 425 citations
  22. True Few-Shot Learning with Language ModelsarXiv:2105.11447Ethan Perez, Douwe Kiela, Kyunghyun Cho2021 · 412 citations
  23. Do Prompt-Based Models Really Understand the Meaning of Their Prompts?Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)Albert Webson, Ellie Pavlick2022 · 407 citations
  24. A Simple Language Model for Task-Oriented DialoguearXiv:2005.00796Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, et al.2020 · 399 citations
  25. Aligning AI With Shared Human ValuesarXiv:2008.02275Dan Hendrycks, Collin Burns, Steven Basart, et al.2020 · 395 citations
  26. FastBERT: a Self-distilling BERT with Adaptive Inference TimeAnnual Meeting of the Association for Computational Linguistics (ACL)Weijie Liu, Peng Zhou, Zhiruo Wang, et al.2020 · 383 citations
  27. Parameter-Efficient Transfer Learning with Diff PruningAnnual Meeting of the Association for Computational Linguistics (ACL)Demi Guo, Alexander Rush, Yoon Kim2021 · 367 citations
  28. Symbolic Knowledge Distillation: from General Language Models to Commonsense ModelsConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Peter West, Chandra Bhagavatula, Jack Hessel, et al.2022 · 349 citations
  29. UL2: Unifying Language Learning ParadigmsarXiv:2205.05131Yi Tay, Mostafa Dehghani, Vinh Q. Tran, et al.2022 · 347 citations
  30. Unsupervised Domain Clusters in Pretrained Language ModelsAnnual Meeting of the Association for Computational Linguistics (ACL)Roee Aharoni, Yoav Goldberg2020 · 330 citations
  31. Carbon Emissions and Large Neural Network TrainingarXiv:2104.10350David Patterson, Joseph Gonzalez, Quoc Le, et al.2021 · 323 citations
  32. ALFWorld: Aligning Text and Embodied Environments for Interactive LearningarXiv:2010.03768Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, et al.2020 · 322 citations
  33. From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP ModelsAnnual Meeting of the Association for Computational Linguistics (ACL)Shangbin Feng, Chan Young Park, Yuhan Liu, et al.2023 · 311 citations
  34. AugGPT: Leveraging ChatGPT for Text Data AugmentationIEEE Transactions on Big DataHaixing Dai; Zhengliang Liu; Wenxiong Liao; Xiaoke Huang; Yihan Cao; Zihao Wu; Lin Zhao; Shaochen Xu; Wei Liu; Ninghao Liu; Sheng Li; Dajiang Zhu; Hongmin Cai; Lichao Sun; Quanzheng Li; Dinggang Shen; Tianming Liu;, Xiang Li2023 · 306 citations
  35. SPoT: Better Frozen Model Adaptation through Soft Prompt TransferAnnual Meeting of the Association for Computational Linguistics (ACL)Tu Vu, Brian Lester, Noah Constant, et al.2022 · 306 citations
  36. ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale TransformersConference on Neural Information Processing Systems (NeurIPS)Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, et al.2022 · 304 citations
  37. On-Policy Distillation of Language Models: Learning from Self-Generated MistakesarXiv:2306.13649Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, et al.2023 · 299 citations
  38. AdapterDrop: On the Efficiency of Adapters in TransformersConference on Empirical Methods in Natural Language Processing (EMNLP)Andreas Rücklé, Gregor Geigle, Max Glockner, et al.2021 · 296 citations
  39. AlignScore: Evaluating Factual Consistency with A Unified Alignment FunctionAnnual Meeting of the Association for Computational Linguistics (ACL)Yuheng Zha, Yichi Yang, Ruichen Li, et al.2023 · 284 citations
  40. Contextualized Perturbation for Textual Adversarial AttackConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Dianqi Li, Yizhe Zhang, Hao Peng, et al.2021 · 283 citations
  41. GPT3Mix: Leveraging Large-scale Language Models for Text AugmentationFindings of the Association for Computational LinguisticsKang Min Yoo, Dongju Park, Jaewook Kang, et al.2021 · 270 citations
  42. DiffCSE: Difference-based Contrastive Learning for Sentence EmbeddingsConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, et al.2022 · 269 citations
  43. ConveRT: Efficient and Accurate Conversational Representations from TransformersFindings of the Association for Computational LinguisticsMatthew Henderson, Iñigo Casanueva, Nikola Mrkšić, et al.2020 · 269 citations
  44. BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval ModelsNeurIPS 2021 Datasets and BenchmarksNandan Thakur, Nils Reimers, Andreas Rücklé, et al.2021 · 265 citations
  45. DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI ScalearXiv:2201.05596Samyam Rajbhandari, Conglong Li, Zhewei Yao, et al.2022 · 261 citations
  46. Multilingual is not enough: BERT for FinnisharXiv:1912.07076Antti Virtanen, Jenna Kanerva, Rami Ilo, et al.2019 · 258 citations
  47. ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and ExplanationAnnual Meeting of the Association for Computational Linguistics (ACL)Vijit Malik, Rishabh Sanjay, Shubham Kumar Nigam, et al.2021 · 251 citations
  48. Interpreting Pretrained Contextualized Representations via Reductions to Static EmbeddingsAnnual Meeting of the Association for Computational Linguistics (ACL)Rishi Bommasani, Kelly Davis, Claire Cardie2020 · 250 citations
  49. A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language ModelsarXiv:2401.01313S.M Towhidul Islam Tonmoy, S M Mehedi Zaman, Vinija Jain, et al.2024 · 249 citations
  50. ZeroGen: Efficient Zero-shot Learning via Dataset GenerationConference on Empirical Methods in Natural Language Processing (EMNLP)Jiacheng Ye, Jiahui Gao, Qintong Li, et al.2022 · 245 citations
  51. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving PerformancearXiv:2305.05176Lingjiao Chen, Matei Zaharia, James Zou2023 · 241 citations
  52. Pretrained Language Models for Biomedical and Clinical Tasks: Understanding and Extending the State-of-the-ArtWorkshop on Clinical Natural Language Processing (Clinical NLP) @ LRECPatrick Lewis, Myle Ott, Jingfei Du, et al.2020 · 241 citations
  53. BERT-of-Theseus: Compressing BERT by Progressive Module ReplacingConference on Empirical Methods in Natural Language Processing (EMNLP)Canwen Xu, Wangchunshu Zhou, Tao Ge, et al.2020 · 240 citations
  54. Structured Pruning Learns Compact and Accurate ModelsAnnual Meeting of the Association for Computational Linguistics (ACL)Mengzhou Xia, Zexuan Zhong, Danqi Chen2022 · 221 citations
  55. Generating Fact Checking ExplanationsAnnual Meeting of the Association for Computational Linguistics (ACL)Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, et al.2020 · 220 citations
  56. TinyStories: How Small Can Language Models Be and Still Speak Coherent English?Ronen Eldan, Yuanzhi Li2023 · 213 citations
  57. TernaryBERT: Distillation-aware Ultra-low Bit BERTConference on Empirical Methods in Natural Language Processing (EMNLP)Wei Zhang, Lu Hou, Yichun Yin, et al.2020 · 213 citations
  58. Large Language Model as Attributed Training Data Generator: A Tale of Diversity and BiasConference on Neural Information Processing Systems (NeurIPS)Yue Yu, Yuchen Zhuang, Jieyu Zhang, et al.2023 · 202 citations
  59. Structured Pruning of Large Language ModelsConference on Empirical Methods in Natural Language Processing (EMNLP)Ziheng Wang, Jeremy Wohlwend, Tao Lei2020 · 202 citations
  60. RRHF: Rank Responses to Align Language Models with Human Feedback without tearsarXiv:2304.05302Zheng Yuan, Hongyi Yuan, Chuanqi Tan, et al.2023 · 201 citations
  61. A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark DatasetsFindings of the Association for Computational LinguisticsMd Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, et al.2023 · 200 citations
  62. Block Pruning For Faster TransformersConference on Empirical Methods in Natural Language Processing (EMNLP)François Lagunas, Ella Charlaix, Victor Sanh, et al.2021 · 199 citations
  63. The Right Tool for the Job: Matching Model and Instance ComplexitiesAnnual Meeting of the Association for Computational Linguistics (ACL)Roy Schwartz, Gabriel Stanovsky, Swabha Swayamdipta, et al.2020 · 196 citations
  64. What Can We Learn from Collective Human Opinions on Natural Language Inference Data?Conference on Empirical Methods in Natural Language Processing (EMNLP)Yixin Nie, Xiang Zhou, Mohit Bansal2020 · 196 citations
  65. Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-TuningInternational Conference on Computational Linguistics (COLING)Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, et al.2022 · 195 citations
  66. HONEST: Measuring Hurtful Sentence Completion in Language ModelsConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Debora Nozza, Federico Bianchi, Dirk Hovy2021 · 191 citations
  67. SemEval-2022 Task 11: Multilingual Complex Named Entity Recognition (MultiCoNER)International Workshop on Semantic Evaluation (SemEval)Shervin Malmasi, Anjie Fang, Besnik Fetahu, et al.2022 · 190 citations
  68. Compressing Large-Scale Transformer-Based Models: A Case Study on BERTTransactions of the Association for Computational Linguistics (TACL)Prakhar Ganesh, Yao Chen, Xin Lou, et al.2021 · 190 citations
  69. Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style TransferConference on Empirical Methods in Natural Language Processing (EMNLP)Fanchao Qi, Yangyi Chen, Xurui Zhang, et al.2021 · 189 citations
  70. Eliciting Latent Predictions from Transformers with the Tuned LensarXiv:2303.08112Nora Belrose, Igor Ostrovsky, Lev McKinney, et al.2023 · 188 citations
  71. SemEval-2023 Task 10: Explainable Detection of Online SexismInternational Workshop on Semantic Evaluation (SemEval)Hannah Kirk, Wenjie Yin, Bertie Vidgen, et al.2023 · 185 citations
  72. Debiasing Pre-trained Contextualised EmbeddingsConference of the European Chapter of the Association for Computational Linguistics (EACL)Masahiro Kaneko, Danushka Bollegala2021 · 175 citations
  73. Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy OptimizationarXiv:2210.01241Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kiante Brantley, et al.2022 · 174 citations
  74. The Troubling Emergence of Hallucination in Large Language Models - An Extensive Definition, Quantification, and Prescriptive RemediationsConference on Empirical Methods in Natural Language Processing (EMNLP)Vipula Rawte, Swagata Chakraborty, Agnibh Pathak, et al.2023 · 171 citations
  75. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning ServingOSDI 2023Zhuohan Li, Lianmin Zheng, Yinmin Zhong, et al.2023 · 170 citations
  76. Quark: Controllable Text Generation with Reinforced UnlearningConference on Neural Information Processing Systems (NeurIPS)Ximing Lu, Sean Welleck, Jack Hessel, et al.2022 · 170 citations
  77. BinaryBERT: Pushing the Limit of BERT QuantizationAnnual Meeting of the Association for Computational Linguistics (ACL)Haoli Bai, Wei Zhang, Lu Hou, et al.2021 · 166 citations
  78. CLIPort: What and Where Pathways for Robotic ManipulationarXiv:2109.12098Mohit Shridhar, Lucas Manuelli, Dieter Fox2021 · 159 citations
  79. GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense RetrievalConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Kexin Wang, Nandan Thakur, Nils Reimers, et al.2022 · 155 citations
  80. AnnoLLM: Making Large Language Models to Be Better Crowdsourced AnnotatorsarXiv:2303.16854Xingwei He, Zhenghao Lin, Yeyun Gong, et al.2023 · 152 citations
  81. Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer LearningFindings of the Association for Computational LinguisticsZhaojiang Lin, Andrea Madotto, Pascale Fung2020 · 152 citations
  82. CPM: A Large-scale Generative Chinese Pre-trained Language ModelAI OpenZhengyan Zhang, Xu Han, Hao Zhou, et al.2020 · 150 citations
  83. Introducing CAD: the Contextual Abuse DatasetConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Bertie Vidgen, Dong Nguyen, Helen Margetts, et al.2021 · 147 citations
  84. Efficient Methods for Natural Language Processing: A SurveyTransactions of the Association for Computational Linguistics (TACL)Marcos Treviso, Ji-Ung Lee, Tianchu Ji, et al.2023 · 145 citations
  85. UNQOVERing Stereotyping Biases via Underspecified QuestionsFindings of the Association for Computational LinguisticsTao Li, Daniel Khashabi, Tushar Khot, et al.2020 · 145 citations
  86. MiniLLM: On-Policy Distillation of Large Language ModelsarXiv:2306.08543Yuxian Gu, Li Dong, Furu Wei, et al.2023 · 144 citations
  87. Identifying and Measuring Annotator Bias Based on Annotators' Demographic CharacteristicsProceedings of the Fourth Workshop on Online Abuse and HarmsHala Al Kuwatly, Maximilian Wich, Georg Groh2020 · 144 citations
  88. Rethinking embedding coupling in pre-trained language modelsarXiv:2010.12821Hyung Won Chung, Thibault Févry, Henry Tsai, et al.2020 · 142 citations
  89. On the Language Neutrality of Pre-trained Multilingual RepresentationsFindings of the Association for Computational LinguisticsJindřich Libovický, Rudolf Rosa, Alexander Fraser2020 · 140 citations
  90. Fantastic Questions and Where to Find Them: FairytaleQA – An Authentic Dataset for Narrative ComprehensionYing Xu, Dakuo Wang, Mo Yu, et al.2022 · 137 citations
  91. Grounded Adaptation for Zero-shot Executable Semantic ParsingConference on Empirical Methods in Natural Language Processing (EMNLP)Victor Zhong, Mike Lewis, Sida I. Wang, et al.2020 · 136 citations
  92. UnNatural Language InferenceAnnual Meeting of the Association for Computational Linguistics (ACL)Koustuv Sinha, Prasanna Parthasarathi, Joelle Pineau, et al.2021 · 135 citations
  93. Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and ApproachesarXiv:1904.01172Shane Storks, Qiaozi Gao, Joyce Y. Chai2019 · 135 citations
  94. Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal DatasetConference on Neural Information Processing Systems (NeurIPS)Peter Henderson, Mark S. Krass, Lucia Zheng, et al.2022 · 133 citations
  95. Learning Compact Metrics for MTConference on Empirical Methods in Natural Language Processing (EMNLP)Amy Pu, Hyung Won Chung, Ankur Parikh, et al.2021 · 130 citations
  96. LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal DomainFindings of the Association for Computational LinguisticsJoel Niklaus, Veton Matoshi, Pooja Rani, et al.2023 · 123 citations
  97. I-BERT: Integer-only BERT QuantizationInternational Conference on Machine Learning (ICML)Sehoon Kim, Amir Gholami, Zhewei Yao, et al.2021 · 122 citations
  98. Logical Fallacy DetectionFindings of the Association for Computational LinguisticsZhijing Jin, Abhinav Lalwani, Tejas Vaidhya, et al.2022 · 119 citations
  99. Unmasking Contextual Stereotypes: Measuring and Mitigating BERT's Gender BiasWorkshop on Gender Bias in Natural Language ProcessingMarion Bartl, Malvina Nissim, Albert Gatt2020 · 116 citations
  100. MoEfication: Transformer Feed-forward Layers are Mixtures of ExpertsFindings of the Association for Computational LinguisticsZhengyan Zhang, Yankai Lin, Zhiyuan Liu, et al.2022 · 115 citations

1 2 3

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sanh-2019-distilbert-distilled-version-bert-smaller/cited-by/