Elamur Research⌕
☰
home / papers / RoBERTa: A Robustly Optimized BERT Pretraining… / cited by

Who cites «RoBERTa: A Robustly Optimized BERT Pretraining Approach»

9,837 papers of our library cite it (of the 149,439 whose bibliographies we read) · 1293 with a page here, the most cited first · page 1 of 13 · 1–100 · ← back to «RoBERTa: A Robustly Optimized BERT Pretraining Approach»

1 2 … 10 … 13

  1. Sentence-BERT: Sentence Embeddings using Siamese BERT-NetworksConference on Empirical Methods in Natural Language Processing (EMNLP)Nils Reimers, Iryna Gurevych2019 · 17,096 citations
  2. Transformers: State-of-the-Art Natural Language ProcessingConference on Empirical Methods in Natural Language Processing (EMNLP)Thomas Wolf, Lysandre Debut, Victor Sanh, et al.2020 · 15,755 citations
  3. Language Models are Few-Shot LearnersarXiv:2005.14165Tom B. Brown, Benjamin Mann, Nick Ryder, et al.2020 · 14,178 citations
  4. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionAnnual Meeting of the Association for Computational Linguistics (ACL)Mike Lewis, Yinhan Liu, Naman Goyal, et al.2020 · 12,856 citations
  5. Exploring the Limits of Transfer Learning with a Unified Text-to-Text TransformerarXiv:1910.10683Colin Raffel, Noam Shazeer, Adam Roberts, et al.2019 · 11,325 citations
  6. Llama 2: Open Foundation and Fine-Tuned Chat ModelsarXiv:2307.09288Hugo Touvron, Louis Martin, Kevin Stone, et al.2023 · 9,669 citations
  7. LoRA: Low-Rank Adaptation of Large Language ModelsarXiv:2106.09685Edward J. Hu, Yelong Shen, Phillip Wallis, et al.2021 · 8,661 citations
  8. The Llama 3 Herd of ModelsarXiv:2407.21783Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al.2024 · 7,682 citations
  9. ChatGPT for Good? On Opportunities and Challenges of Large Language Models for EducationEnkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, et al.2023 · 7,440 citations
  10. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language ProcessingACM Computing SurveysPengfei Liu, Weizhe Yuan, Jinlan Fu, et al.2021 · 5,423 citations
  11. SimCSE: Simple Contrastive Learning of Sentence EmbeddingsConference on Empirical Methods in Natural Language Processing (EMNLP)Tianyu Gao, Xingcheng Yao, Danqi Chen2021 · 3,942 citations
  12. Longformer: The Long-Document TransformerarXiv:2004.05150Iz Beltagy, Matthew E. Peters, Arman Cohan2020 · 3,880 citations
  13. XLNet: Generalized Autoregressive Pretraining for Language UnderstandingarXiv:1906.08237Zhilin Yang, Zihang Dai, Yiming Yang, et al.2019 · 3,857 citations
  14. Measuring Massive Multitask Language UnderstandingarXiv:2009.03300Dan Hendrycks, Collin Burns, Steven Basart, et al.2020 · 3,620 citations
  15. HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden UnitsIEEE/ACM Transactions on Audio Speech and Language ProcessingWei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, et al.2021 · 3,458 citations
  16. Scaling Laws for Neural Language ModelsarXiv:2001.08361Jared Kaplan, Sam McCandlish, Tom Henighan, et al.2020 · 3,434 citations
  17. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open QuestionsACM Transactions on Information SystemsLei Huang, Weijiang Yu, Weitao Ma, et al.2023 · 3,231 citations
  18. mT5: A Massively Multilingual Pre-trained Text-to-Text TransformerConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Linting Xue, Noah Constant, Adam Roberts, et al.2021 · 3,226 citations
  19. On the Opportunities and Risks of Foundation ModelsarXiv:2108.07258Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, et al.2021 · 3,160 citations
  20. CodeBERT: A Pre-Trained Model for Programming and Natural LanguagesFindings of the Association for Computational LinguisticsZhangyin Feng, Daya Guo, Duyu Tang, et al.2020 · 2,967 citations
  21. QLoRA: Efficient Finetuning of Quantized LLMsConference on Neural Information Processing Systems (NeurIPS)Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, et al.2023 · 2,929 citations
  22. A Survey of Large Language ModelsFrontiers of Computer ScienceWayne Xin Zhao, Kun Zhou, Junyi Li, et al.2023 · 2,875 citations
  23. ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsarXiv:1909.11942Zhenzhong Lan, Mingda Chen, Sebastian Goodman, et al.2019 · 2,806 citations
  24. Domain-Specific Language Model Pretraining for Biomedical Natural Language ProcessingarXiv:2007.15779Yu Gu; Robert Tinn; Hao Cheng; Michael Lucas; Naoto Usuyama; Xiaodong Liu; Tristan Naumann; Jianfeng Gao;, Hoifung Poon2020 · 2,674 citations
  25. Evaluating Large Language Models Trained on CodearXiv:2107.03374Mark Chen, Jerry Tworek, Heewoo Jun, et al.2021 · 2,659 citations
  26. SpanBERT: Improving Pre-training by Representing and Predicting SpansTransactions of the Association for Computational Linguistics (TACL)Mandar Joshi, Danqi Chen, Yinhan Liu, et al.2019 · 2,616 citations
  27. UNITER: UNiversal Image-TExt Representation LearningLecture notes in computer scienceYen-Chun Chen, Linjie Li, Licheng Yu, et al.2019 · 2,418 citations
  28. TinyBERT: Distilling BERT for Natural Language UnderstandingFindings of the Association for Computational LinguisticsXiaoqi Jiao, Yichun Yin, Lifeng Shang, et al.2020 · 2,378 citations
  29. BLEURT: Learning Robust Metrics for Text GenerationAnnual Meeting of the Association for Computational Linguistics (ACL)Thibault Sellam, Dipanjan Das, Ankur Parikh2020 · 2,032 citations
  30. Making Pre-trained Language Models Better Few-shot LearnersAnnual Meeting of the Association for Computational Linguistics (ACL)Tianyu Gao, Adam Fisch, Danqi Chen2021 · 1,987 citations
  31. On Faithfulness and Factuality in Abstractive SummarizationAnnual Meeting of the Association for Computational Linguistics (ACL)Joshua Maynez, Shashi Narayan, Bernd Bohnet, et al.2020 · 1,857 citations
  32. DeBERTa: Decoding-enhanced BERT with Disentangled AttentionarXiv:2006.03654Pengcheng He, Xiaodong Liu, Jianfeng Gao, et al.2020 · 1,828 citations
  33. Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language InferenceConference of the European Chapter of the Association for Computational Linguistics (EACL)Timo Schick, Hinrich Schütze2021 · 1,817 citations
  34. Pre-trained Models for Natural Language Processing: A SurveyarXiv:2003.08271 · SCIENCE CHINA Technological Sciences,2020, 63, 1872-1897Xipeng Qiu, Tianxiang Sun, Yige Xu, et al.2020 · 1,759 citations
  35. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelsarXiv:2206.04615 · Transactions on Machine Learning Research, May/2022, https://openreview.net/forum?id=uyTL5BvosjAarohi Srivastava, Abhinav Rastogi, Abhishek Rao, et al.2022 · 1,756 citations
  36. OPT: Open Pre-trained Transformer Language ModelsarXiv:2205.01068Susan Zhang, Stephen Roller, Naman Goyal, et al.2022 · 1,755 citations
  37. BERTopic: Neural topic modeling with a class-based TF-IDF procedurearXiv:2203.05794Maarten Grootendorst2022 · 1,737 citations
  38. Qwen Technical ReportarXiv:2309.16609Jinze Bai, Shuai Bai, Yunfei Chu, et al.2023 · 1,669 citations
  39. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text SupervisionInternational Conference on Machine Learning (ICML)Chao Jia, Yinfei Yang, Ye Xia, et al.2021 · 1,615 citations
  40. REALM: Retrieval-Augmented Language Model Pre-TrainingarXiv:2002.08909Kelvin Guu, Kenton Lee, Zora Tung, et al.2020 · 1,580 citations
  41. Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A SurveyACM Computing SurveysBonan Min, Hayley Ross, Elior Sulem, et al.2021 · 1,556 citations
  42. Unifying Large Language Models and Knowledge Graphs: A RoadmapIEEE Transactions on Knowledge and Data EngineeringShirui Pan, Linhao Luo, Yufei Wang, et al.2023 · 1,551 citations
  43. LEGAL-BERT: The Muppets straight out of Law SchoolFindings of the Association for Computational LinguisticsIlias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, et al.2020 · 1,493 citations
  44. Pre-Training with Whole Word Masking for Chinese BERTIEEE/ACM Transactions on Audio Speech and Language ProcessingYiming Cui, Wanxiang Che, Ting Liu, et al.2019 · 1,490 citations
  45. GLM: General Language Model Pretraining with Autoregressive Blank InfillingAnnual Meeting of the Association for Computational Linguistics (ACL)Zhengxiao Du, Yujie Qian, Xiao Liu, et al.2022 · 1,455 citations
  46. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?Conference on Empirical Methods in Natural Language Processing (EMNLP)Sewon Min, Xinxi Lyu, Ari Holtzman, et al.2022 · 1,444 citations
  47. Code Llama: Open Foundation Models for CodearXiv:2308.12950Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, et al.2024 · 1,430 citations
  48. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech RepresentationsConference on Neural Information Processing Systems (NeurIPS)Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, et al.2020 · 1,421 citations
  49. LaMDA: Language Models for Dialog ApplicationsarXiv:2201.08239Romal Thoppilan, Daniel De Freitas, Jamie Hall, et al.2022 · 1,344 citations
  50. DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding SharingarXiv:2111.09543Pengcheng He, Jianfeng Gao, Weizhu Chen2021 · 1,320 citations
  51. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language ModelsFindings of the Association for Computational LinguisticsSamuel Gehman, Suchin Gururangan, Maarten Sap, et al.2020 · 1,306 citations
  52. AraBERT: Transformer-based Model for Arabic Language UnderstandingarXiv:2003.00104Wissam Antoun, Fady Baly, Hazem Hajj2020 · 1,304 citations
  53. Adversarial NLI: A New Benchmark for Natural Language UnderstandingAnnual Meeting of the Association for Computational Linguistics (ACL)Yixin Nie, Adina Williams, Emily Dinan, et al.2020 · 1,296 citations
  54. A Comprehensive Overview of Large Language ModelsACM Transactions on Intelligent Systems and TechnologyHumza Naveed, Asad Ullah Khan, Shi Qiu, et al.2023 · 1,287 citations
  55. CamemBERT: a Tasty French Language ModelAnnual Meeting of the Association for Computational Linguistics (ACL)Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, et al.2019 · 1,269 citations
  56. PIQA: Reasoning about Physical Commonsense in Natural LanguageProceedings of the AAAI Conference on Artificial IntelligenceYonatan Bisk, Rowan Zellers, Ronan Le Bras, et al.2019 · 1,264 citations
  57. What Makes Good In-Context Examples for GPT-3?Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning ArchitecturesJiachang Liu, Dinghan Shen, Yizhe Zhang, et al.2022 · 1,256 citations
  58. Recipes for Building an Open-Domain ChatbotConference of the European Chapter of the Association for Computational Linguistics (EACL)Stephen Roller, Emily Dinan, Naman Goyal, et al.2021 · 1,214 citations
  59. Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order SensitivityAnnual Meeting of the Association for Computational Linguistics (ACL)Yao Lu, Max Bartolo, Alastair Moore, et al.2022 · 1,209 citations
  60. BERTweet: A pre-trained language model for English TweetsConference on Empirical Methods in Natural Language Processing (EMNLP)Dat Quoc Nguyen, Thanh Vu, Anh Tuan Nguyen2020 · 1,144 citations
  61. StereoSet: Measuring stereotypical bias in pretrained language modelsAnnual Meeting of the Association for Computational Linguistics (ACL)Moin Nadeem, Anna Bethke, Siva Reddy2021 · 1,141 citations
  62. It's Not Just Size That Matters: Small Language Models Are Also Few-Shot LearnersConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Timo Schick, Hinrich Schütze2021 · 1,132 citations
  63. Bias and Fairness in Large Language Models: A SurveyComputational LinguisticsIsabel O. Gallegos, Ryan A. Rossi, Joe Barrow, et al.2023 · 1,082 citations
  64. VL-BERT: Pre-training of Generic Visual-Linguistic RepresentationsarXiv:1908.08530Weijie Su, Xizhou Zhu, Yue Cao, et al.2019 · 1,066 citations
  65. GPT Understands, TooAI OpenXiao Liu, Yanan Zheng, Zhengxiao Du, et al.2021 · 1,054 citations
  66. Big Bird: Transformers for Longer SequencesConference on Neural Information Processing Systems (NeurIPS)Manzil Zaheer, Guru Guruganesh, Avinava Dubey, et al.2020 · 1,023 citations
  67. K-BERT: Enabling Language Representation with Knowledge GraphProceedings of the AAAI Conference on Artificial IntelligenceWeijie Liu, Peng Zhou, Zhe Zhao, et al.2019 · 1,019 citations
  68. GPQA: A Graduate-Level Google-Proof Q&A BenchmarkarXiv:2311.12022David Rein, Betty Li Hou, Asa Cooper Stickland, et al.2023 · 1,018 citations
  69. P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and TasksAnnual Meeting of the Association for Computational Linguistics (ACL)Xiao Liu, Kaixuan Ji, Yicheng Fu, et al.2022 · 1,017 citations
  70. MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited DevicesAnnual Meeting of the Association for Computational Linguistics (ACL)Zhiqing Sun, Hongkun Yu, Xiaodan Song, et al.2020 · 1,016 citations
  71. Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language ModelsComputational LinguisticsYue Zhang, Yafu Li, Leyang Cui, et al.2023 · 977 citations
  72. AdapterFusion: Non-Destructive Task Composition for Transfer LearningarXiv:2005.00247 · Proceedings of EACL 2021Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, et al.2020 · 958 citations
  73. MPNet: Masked and Permuted Pre-training for Language UnderstandingarXiv:2004.09297Kaitao Song, Xu Tan, Tao Qin, et al.2020 · 957 citations
  74. TweetEval: Unified Benchmark and Comparative Evaluation for Tweet ClassificationarXiv:2010.12421Francesco Barbieri, Jose Camacho-Collados, Leonardo Neves, et al.2020 · 956 citations
  75. Extracting Training Data from Large Language ModelsarXiv:2012.07805Nicholas Carlini, Florian Tramer, Eric Wallace, et al.2020 · 955 citations
  76. Patient Knowledge Distillation for BERT Model CompressionarXiv:1908.09355Siqi Sun, Yu Cheng, Zhe Gan, et al.2019 · 947 citations
  77. C-Pack: Packed Resources For General Chinese EmbeddingsInternational ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)Shitao Xiao, Zheng Liu, Peitian Zhang, et al.2023 · 922 citations
  78. CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language ModelsConference on Empirical Methods in Natural Language Processing (EMNLP)Nikita Nangia, Clara Vania, Rasika Bhalerao, et al.2020 · 897 citations
  79. LayoutLM: Pre-training of Text and Layout for Document Image UnderstandingarXiv:1912.13318Yiheng Xu, Minghao Li, Lei Cui, et al.2019 · 896 citations
  80. Are NLP Models really able to Solve Simple Math Word Problems?Arkil Patel, Satwik Bhattamishra, Navin Goyal2021 · 890 citations
  81. UNIFIEDQA: Crossing Format Boundaries with a Single QA SystemFindings of the Association for Computational LinguisticsDaniel Khashabi, Sewon Min, Tushar Khot, et al.2020 · 857 citations
  82. TaPas: Weakly Supervised Table Parsing via Pre-trainingAnnual Meeting of the Association for Computational Linguistics (ACL)Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, et al.2020 · 847 citations
  83. On the Sentence Embeddings from Pre-trained Language ModelsConference on Empirical Methods in Natural Language Processing (EMNLP)Bohan Li, Hao Zhou, Junxian He, et al.2020 · 833 citations
  84. Is ChatGPT a General-Purpose Natural Language Processing Task Solver?Conference on Empirical Methods in Natural Language Processing (EMNLP)Chengwei Qin, Aston Zhang, Zhuosheng Zhang, et al.2023 · 828 citations
  85. Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and BeyondACM Transactions on Knowledge Discovery from DataJingfeng Yang, Hongye Jin, Ruixiang Tang, et al.2023 · 814 citations
  86. In-Context Retrieval-Augmented Language ModelsTransactions of the Association for Computational Linguistics (TACL)Ori Ram, Yoav Levine, Itay Dalmedigos, et al.2023 · 793 citations
  87. Can Large Language Models Transform Computational Social Science?Computational LinguisticsCaleb Ziems, William Held, Omar Shaikh, et al.2023 · 790 citations
  88. Explainability for Large Language Models: A SurveyACM Transactions on Intelligent Systems and TechnologyHaiyan Zhao, Hanjie Chen, Fan Yang, et al.2023 · 790 citations
  89. Revisiting Pre-Trained Models for Chinese Natural Language ProcessingarXiv:2004.13922Yiming Cui, Wanxiang Che, Ting Liu, et al.2020 · 790 citations
  90. Towards Reasoning in Large Language Models: A SurveyFindings of the Association for Computational LinguisticsJie Huang, Kevin Chen-Chuan Chang2023 · 780 citations
  91. LayoutLMv3: Pre-training for Document AI with Unified Text and Image MaskingACM International Conference on Multimedia (ACM MM)Yupan Huang, Tengchao Lv, Lei Cui, et al.2022 · 778 citations
  92. AdapterHub: A Framework for Adapting TransformersConference on Empirical Methods in Natural Language Processing (EMNLP)Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, et al.2020 · 754 citations
  93. Unified Pre-training for Program Understanding and GenerationConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, et al.2021 · 750 citations
  94. Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning StepsInternational Conference on Computational Linguistics (COLING)Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, et al.2020 · 741 citations
  95. IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian LanguagesFindings of the Association for Computational LinguisticsDivyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, et al.2020 · 732 citations
  96. Fact or Fiction: Verifying Scientific ClaimsConference on Empirical Methods in Natural Language Processing (EMNLP)David Wadden, Shanchuan Lin, Kyle Lo, et al.2020 · 709 citations
  97. SPECTER: Document-level Representation Learning using Citation-informed TransformersAnnual Meeting of the Association for Computational Linguistics (ACL)Arman Cohan, Sergey Feldman, Iz Beltagy, et al.2020 · 697 citations
  98. CORD-19: The COVID-19 Open Research DatasetWorkshop on NLP for COVID-19 at ACLLucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, et al.2020 · 695 citations
  99. TaBERT: Pretraining for Joint Understanding of Textual and Tabular DataAnnual Meeting of the Association for Computational Linguistics (ACL)Pengcheng Yin, Graham Neubig, Wen-tau Yih, et al.2020 · 692 citations
  100. Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text RetrievalarXiv:2007.00808Lee Xiong, Chenyan Xiong, Ye Li, et al.2020 · 688 citations

1 2 … 10 … 13

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/cited-by/