Elamur Research⌕
☰
home / papers / ALBERT: A Lite BERT for Self-supervised Learning… / cited by

Who cites «ALBERT: A Lite BERT for Self-supervised Learning of Language Representations»

1,861 papers of our library cite it (of the 149,439 whose bibliographies we read) · 262 with a page here, the most cited first · page 1 of 3 · 1–100 · ← back to «ALBERT: A Lite BERT for Self-supervised Learning of Language Representations»

1 2 3

  1. Transformers: State-of-the-Art Natural Language ProcessingConference on Empirical Methods in Natural Language Processing (EMNLP)Thomas Wolf, Lysandre Debut, Victor Sanh, et al.2020 · 15,755 citations
  2. Language Models are Few-Shot LearnersarXiv:2005.14165Tom B. Brown, Benjamin Mann, Nick Ryder, et al.2020 · 14,178 citations
  3. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionAnnual Meeting of the Association for Computational Linguistics (ACL)Mike Lewis, Yinhan Liu, Naman Goyal, et al.2020 · 12,856 citations
  4. Exploring the Limits of Transfer Learning with a Unified Text-to-Text TransformerarXiv:1910.10683Colin Raffel, Noam Shazeer, Adam Roberts, et al.2019 · 11,325 citations
  5. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language ProcessingACM Computing SurveysPengfei Liu, Weizhe Yuan, Jinlan Fu, et al.2021 · 5,423 citations
  6. XLNet: Generalized Autoregressive Pretraining for Language UnderstandingarXiv:1906.08237Zhilin Yang, Zihang Dai, Yiming Yang, et al.2019 · 3,857 citations
  7. Measuring Massive Multitask Language UnderstandingarXiv:2009.03300Dan Hendrycks, Collin Burns, Steven Basart, et al.2020 · 3,620 citations
  8. Scaling Laws for Neural Language ModelsarXiv:2001.08361Jared Kaplan, Sam McCandlish, Tom Henighan, et al.2020 · 3,434 citations
  9. On the Opportunities and Risks of Foundation ModelsarXiv:2108.07258Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, et al.2021 · 3,160 citations
  10. RoFormer: Enhanced Transformer with Rotary Position EmbeddingarXiv:2104.09864Jianlin Su, Yu Lu, Shengfeng Pan, et al.2021 · 3,028 citations
  11. UNITER: UNiversal Image-TExt Representation LearningLecture notes in computer scienceYen-Chun Chen, Linjie Li, Licheng Yu, et al.2019 · 2,418 citations
  12. TinyBERT: Distilling BERT for Natural Language UnderstandingFindings of the Association for Computational LinguisticsXiaoqi Jiao, Yichun Yin, Lifeng Shang, et al.2020 · 2,378 citations
  13. DeBERTa: Decoding-enhanced BERT with Disentangled AttentionarXiv:2006.03654Pengcheng He, Xiaodong Liu, Jianfeng Gao, et al.2020 · 1,828 citations
  14. Pre-trained Models for Natural Language Processing: A SurveyarXiv:2003.08271 · SCIENCE CHINA Technological Sciences,2020, 63, 1872-1897Xipeng Qiu, Tianxiang Sun, Yige Xu, et al.2020 · 1,759 citations
  15. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelsarXiv:2206.04615 · Transactions on Machine Learning Research, May/2022, https://openreview.net/forum?id=uyTL5BvosjAarohi Srivastava, Abhinav Rastogi, Abhishek Rao, et al.2022 · 1,756 citations
  16. BERTopic: Neural topic modeling with a class-based TF-IDF procedurearXiv:2203.05794Maarten Grootendorst2022 · 1,737 citations
  17. Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A SurveyACM Computing SurveysBonan Min, Hayley Ross, Elior Sulem, et al.2021 · 1,556 citations
  18. Unifying Large Language Models and Knowledge Graphs: A RoadmapIEEE Transactions on Knowledge and Data EngineeringShirui Pan, Linhao Luo, Yufei Wang, et al.2023 · 1,551 citations
  19. Efficient Transformers: A SurveyACM Computing SurveysYi Tay, Mostafa Dehghani, Dara Bahri, et al.2020 · 1,539 citations
  20. LEGAL-BERT: The Muppets straight out of Law SchoolFindings of the Association for Computational LinguisticsIlias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, et al.2020 · 1,493 citations
  21. Pre-Training with Whole Word Masking for Chinese BERTIEEE/ACM Transactions on Audio Speech and Language ProcessingYiming Cui, Wanxiang Che, Ting Liu, et al.2019 · 1,490 citations
  22. GLM: General Language Model Pretraining with Autoregressive Blank InfillingAnnual Meeting of the Association for Computational Linguistics (ACL)Zhengxiao Du, Yujie Qian, Xiao Liu, et al.2022 · 1,455 citations
  23. LaMDA: Language Models for Dialog ApplicationsarXiv:2201.08239Romal Thoppilan, Daniel De Freitas, Jamie Hall, et al.2022 · 1,344 citations
  24. DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding SharingarXiv:2111.09543Pengcheng He, Jianfeng Gao, Weizhu Chen2021 · 1,320 citations
  25. AraBERT: Transformer-based Model for Arabic Language UnderstandingarXiv:2003.00104Wissam Antoun, Fady Baly, Hazem Hajj2020 · 1,304 citations
  26. CamemBERT: a Tasty French Language ModelAnnual Meeting of the Association for Computational Linguistics (ACL)Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, et al.2019 · 1,269 citations
  27. It's Not Just Size That Matters: Small Language Models Are Also Few-Shot LearnersConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Timo Schick, Hinrich Schütze2021 · 1,132 citations
  28. Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language ModelsComputational LinguisticsYue Zhang, Yafu Li, Leyang Cui, et al.2023 · 977 citations
  29. CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language ModelsConference on Empirical Methods in Natural Language Processing (EMNLP)Nikita Nangia, Clara Vania, Rasika Bhalerao, et al.2020 · 897 citations
  30. UNIFIEDQA: Crossing Format Boundaries with a Single QA SystemFindings of the Association for Computational LinguisticsDaniel Khashabi, Sewon Min, Tushar Khot, et al.2020 · 857 citations
  31. Unsupervised Dense Information Retrieval with Contrastive LearningarXiv:2112.09118Gautier Izacard, Mathilde Caron, Lucas Hosseini, et al.2021 · 836 citations
  32. Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and BeyondACM Transactions on Knowledge Discovery from DataJingfeng Yang, Hongye Jin, Ruixiang Tang, et al.2023 · 814 citations
  33. Revisiting Pre-Trained Models for Chinese Natural Language ProcessingarXiv:2004.13922Yiming Cui, Wanxiang Che, Ting Liu, et al.2020 · 790 citations
  34. IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian LanguagesFindings of the Association for Computational LinguisticsDivyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, et al.2020 · 732 citations
  35. CCNet: Extracting High Quality Monolingual Datasets from Web Crawl DataLanguage Resources and Evaluation Conference (LREC)Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, et al.2020 · 666 citations
  36. A Frustratingly Easy Approach for Entity and Relation ExtractionConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Zexuan Zhong, Danqi Chen2020 · 631 citations
  37. A Robustly Optimized BERT Pre-training Approach with Post-trainingChinese National Conference on Computational LinguisticsLiu Zhuang, Lin Wayne, Shi Ya, et al.2021 · 587 citations
  38. What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical ExamsarXiv:2009.13081Di Jin, Eileen Pan, Nassim Oufattole, et al.2020 · 567 citations
  39. Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-TuningAnnual Meeting of the Association for Computational Linguistics (ACL)Armen Aghajanyan, Sonal Gupta, Luke Zettlemoyer2021 · 542 citations
  40. DeCLUTR: Deep Contrastive Learning for Unsupervised Textual RepresentationsAnnual Meeting of the Association for Computational Linguistics (ACL)John Giorgi, Osvald Nitski, Bo Wang, et al.2021 · 516 citations
  41. HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingConference on Empirical Methods in Natural Language Processing (EMNLP)Linjie Li, Yen-Chun Chen, Yu Cheng, et al.2020 · 481 citations
  42. ARBERT & MARBERT: Deep Bidirectional Transformers for ArabicarXiv:2101.01785 · ACL-2021 camera ready versionMuhammad Abdul-Mageed, AbdelRahim Elmadany, El Moatez Billah Nagoudi2021 · 478 citations
  43. Large Language Models: A SurveyarXiv:2402.06196Shervin Minaee, Tomas Mikolov, Narjes Nikzad, et al.2024 · 471 citations
  44. LLM-Pruner: On the Structural Pruning of Large Language ModelsConference on Neural Information Processing Systems (NeurIPS)Xinyin Ma, Gongfan Fang, Xinchao Wang2023 · 452 citations
  45. Pretrained Transformers Improve Out-of-Distribution RobustnessAnnual Meeting of the Association for Computational Linguistics (ACL)Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, et al.2020 · 452 citations
  46. FreeLB: Enhanced Adversarial Training for Natural Language UnderstandingarXiv:1909.11764Chen Zhu, Yu Cheng, Zhe Gan, et al.2019 · 452 citations
  47. Experience Grounds LanguageConference on Empirical Methods in Natural Language Processing (EMNLP)Yonatan Bisk, Ari Holtzman, Jesse Thomason, et al.2020 · 439 citations
  48. GPT-NER: Named Entity Recognition via Large Language ModelsFindings of the Association for Computational LinguisticsShuhe Wang, Xiaofei Sun, Xiaoya Li, et al.2025 · 436 citations
  49. Factual Probing Is [MASK]: Learning vs. Learning to RecallConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Zexuan Zhong, Dan Friedman, Danqi Chen2021 · 415 citations
  50. True Few-Shot Learning with Language ModelsarXiv:2105.11447Ethan Perez, Douwe Kiela, Kyunghyun Cho2021 · 412 citations
  51. Do Prompt-Based Models Really Understand the Meaning of Their Prompts?Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)Albert Webson, Ellie Pavlick2022 · 407 citations
  52. Aligning AI With Shared Human ValuesarXiv:2008.02275Dan Hendrycks, Collin Burns, Steven Basart, et al.2020 · 395 citations
  53. UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-TrainingarXiv:2002.12804Hangbo Bao, Li Dong, Furu Wei, et al.2020 · 395 citations
  54. ETC: Encoding Long and Structured Inputs in TransformersConference on Empirical Methods in Natural Language Processing (EMNLP)Joshua Ainslie, Santiago Ontañón, Chris Alberti, et al.2020 · 386 citations
  55. German's Next Language ModelInternational Conference on Computational Linguistics (COLING)Branden Chan, Stefan Schweter, Timo Möller2020 · 385 citations
  56. FastBERT: a Self-distilling BERT with Adaptive Inference TimeAnnual Meeting of the Association for Computational Linguistics (ACL)Weijie Liu, Peng Zhou, Zhiruo Wang, et al.2020 · 383 citations
  57. Revisiting Few-sample BERT Fine-tuningarXiv:2006.05987Tianyi Zhang, Felix Wu, Arzoo Katiyar, et al.2020 · 368 citations
  58. Parameter-Efficient Transfer Learning with Diff PruningAnnual Meeting of the Association for Computational Linguistics (ACL)Demi Guo, Alexander Rush, Yoon Kim2021 · 367 citations
  59. Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text ClassificationAnnual Meeting of the Association for Computational Linguistics (ACL)Shengding Hu, Ning Ding, Huadong Wang, et al.2022 · 362 citations
  60. BERTje: A Dutch BERT ModelarXiv:1912.09582Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, et al.2019 · 361 citations
  61. RobBERT: a Dutch RoBERTa-based Language ModelarXiv:2001.06286Pieter Delobelle, Thomas Winters, Bettina Berendt2020 · 349 citations
  62. ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and GenerationarXiv:2107.02137Yu Sun, Shuohuan Wang, Shikun Feng, et al.2021 · 332 citations
  63. Unsupervised Corpus Aware Language Model Pre-training for Dense Passage RetrievalAnnual Meeting of the Association for Computational Linguistics (ACL)Luyu Gao, Jamie Callan2022 · 328 citations
  64. How many data points is a prompt worth?Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)Teven Le Scao, Alexander Rush2021 · 320 citations
  65. From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP ModelsAnnual Meeting of the Association for Computational Linguistics (ACL)Shangbin Feng, Chan Young Park, Yuhan Liu, et al.2023 · 311 citations
  66. Byte Pair Encoding is Suboptimal for Language Model PretrainingFindings of the Association for Computational LinguisticsKaj Bostrom, Greg Durrett2020 · 310 citations
  67. SPoT: Better Frozen Model Adaptation through Soft Prompt TransferAnnual Meeting of the Association for Computational Linguistics (ACL)Tu Vu, Brian Lester, Noah Constant, et al.2022 · 306 citations
  68. ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale TransformersConference on Neural Information Processing Systems (NeurIPS)Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, et al.2022 · 304 citations
  69. SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized OptimizationAnnual Meeting of the Association for Computational Linguistics (ACL)Haoming Jiang, Pengcheng He, Weizhu Chen, et al.2020 · 304 citations
  70. Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question AnsweringConference on Empirical Methods in Natural Language Processing (EMNLP)Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, et al.2020 · 300 citations
  71. FlauBERT: Unsupervised Language Model Pre-training for FrencharXiv:1912.05372Hang Le, Loïc Vial, Jibril Frej, et al.2019 · 299 citations
  72. BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in BanglaFindings of the Association for Computational LinguisticsAbhik Bhattacharjee, Tahmid Hasan, Wasi Ahmad, et al.2022 · 296 citations
  73. AdapterDrop: On the Efficiency of Adapters in TransformersConference on Empirical Methods in Natural Language Processing (EMNLP)Andreas Rücklé, Gregor Geigle, Max Glockner, et al.2021 · 296 citations
  74. PAQ: 65 Million Probably-Asked Questions and What You Can Do With ThemTransactions of the Association for Computational Linguistics (TACL)Patrick Lewis, Yuxiang Wu, Linqing Liu, et al.2021 · 288 citations
  75. Two are Better than One: Joint Entity and Relation Extraction with Table-Sequence EncodersConference on Empirical Methods in Natural Language Processing (EMNLP)Jue Wang, Wei Lu2020 · 288 citations
  76. Spelling Error Correction with Soft-Masked BERTAnnual Meeting of the Association for Computational Linguistics (ACL)Shaohua Zhang, Haoran Huang, Jicong Liu, et al.2020 · 284 citations
  77. TRUE: Re-evaluating Factual Consistency EvaluationConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Or Honovich, Roee Aharoni, Jonathan Herzig, et al.2022 · 279 citations
  78. Text Classification Using Label Names Only: A Language Model Self-Training ApproachConference on Empirical Methods in Natural Language Processing (EMNLP)Yu Meng, Yunyi Zhang, Jiaxin Huang, et al.2020 · 274 citations
  79. Attention is Not Only a Weight: Analyzing Transformers with Vector NormsConference on Empirical Methods in Natural Language Processing (EMNLP)Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, et al.2020 · 270 citations
  80. Perceiver: General Perception with Iterative AttentionarXiv:2103.03206Andrew Jaegle, Felix Gimeno, Andrew Brock, et al.2021 · 269 citations
  81. ConveRT: Efficient and Accurate Conversational Representations from TransformersFindings of the Association for Computational LinguisticsMatthew Henderson, Iñigo Casanueva, Nikola Mrkšić, et al.2020 · 269 citations
  82. Injecting Numerical Reasoning Skills into Language ModelsAnnual Meeting of the Association for Computational Linguistics (ACL)Mor Geva, Ankit Gupta, Jonathan Berant2020 · 269 citations
  83. CUAD: An Expert-Annotated NLP Dataset for Legal Contract ReviewarXiv:2103.06268Dan Hendrycks, Collin Burns, Anya Chen, et al.2021 · 268 citations
  84. IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language UnderstandingConference of the Asia-Pacific Chapter of the Association for Computational Linguistics (AACL)Bryan Wilie, Karissa Vincentio, Genta Indra Winata, et al.2020 · 266 citations
  85. DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI ScalearXiv:2201.05596Samyam Rajbhandari, Conglong Li, Zhewei Yao, et al.2022 · 261 citations
  86. Condenser: a Pre-training Architecture for Dense RetrievalConference on Empirical Methods in Natural Language Processing (EMNLP)Luyu Gao, Jamie Callan2021 · 261 citations
  87. SemEval-2020 Task 8: Memotion Analysis- the Visuo-Lingual Metaphor!International Workshop on Semantic Evaluation (SemEval)Chhavi Sharma, Deepesh Bhageria, William Scott, et al.2020 · 261 citations
  88. CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive SummarizationConference on Empirical Methods in Natural Language Processing (EMNLP)Shuyang Cao, Lu Wang2021 · 260 citations
  89. Measuring and Reducing Gendered Correlations in Pre-trained ModelsarXiv:2010.06032Kellie Webster, Xuezhi Wang, Ian Tenney, et al.2020 · 254 citations
  90. GPT-RE: In-context Learning for Relation Extraction using Large Language ModelsConference on Empirical Methods in Natural Language Processing (EMNLP)Zhen Wan, Fei Cheng, Zhuoyuan Mao, et al.2023 · 253 citations
  91. A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language ModelsarXiv:2401.01313S.M Towhidul Islam Tonmoy, S M Mehedi Zaman, Vinija Jain, et al.2024 · 249 citations
  92. Learning Span-Level Interactions for Aspect Sentiment Triplet ExtractionAnnual Meeting of the Association for Computational Linguistics (ACL)Lu Xu, Yew Ken Chia, Lidong Bing2021 · 246 citations
  93. ZeroGen: Efficient Zero-shot Learning via Dataset GenerationConference on Empirical Methods in Natural Language Processing (EMNLP)Jiacheng Ye, Jiahui Gao, Qintong Li, et al.2022 · 245 citations
  94. CLUE: A Chinese Language Understanding Evaluation BenchmarkarXiv:2004.05986Liang Xu, Hai Hu, Xuanwei Zhang, et al.2020 · 244 citations
  95. Hierarchical Graph Network for Multi-hop Question AnsweringConference on Empirical Methods in Natural Language Processing (EMNLP)Yuwei Fang, Siqi Sun, Zhe Gan, et al.2020 · 243 citations
  96. BERT-of-Theseus: Compressing BERT by Progressive Module ReplacingConference on Empirical Methods in Natural Language Processing (EMNLP)Canwen Xu, Wangchunshu Zhou, Tao Ge, et al.2020 · 240 citations
  97. How Context Affects Language Models' Factual PredictionsarXiv:2005.04611Fabio Petroni, Patrick Lewis, Aleksandra Piktus, et al.2020 · 234 citations
  98. A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource LanguagesAnnual Meeting of the Association for Computational Linguistics (ACL)Pedro Javier Ortiz Suárez, Laurent Romary, Benoît Sagot2020 · 222 citations
  99. What Happens To BERT Embeddings During Fine-tuning?BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLPAmil Merchant, Elahe Rahimtoroghi, Ellie Pavlick, et al.2020 · 220 citations
  100. Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language ModelsFindings of the Association for Computational LinguisticsRobert Logan IV, Ivana Balazevic, Eric Wallace, et al.2022 · 216 citations

1 2 3

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lan-2019-albert-lite-bert-self-supervised/cited-by/