Elamur Research⌕
☰
home / papers / RoBERTa: A Robustly Optimized BERT Pretraining… / cited by / page 2

Who cites «RoBERTa: A Robustly Optimized BERT Pretraining Approach»

9,837 papers of our library cite it (of the 149,439 whose bibliographies we read) · 1406 with a page here, the most cited first · page 2 of 15 · 101–200 · ← back to «RoBERTa: A Robustly Optimized BERT Pretraining Approach»

1 2 3 … 10 … 15

  1. TaBERT: Pretraining for Joint Understanding of Textual and Tabular DataAnnual Meeting of the Association for Computational Linguistics (ACL)Pengcheng Yin, Graham Neubig, Wen-tau Yih, et al.2020 · 692 citations
  2. Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text RetrievalarXiv:2007.00808Lee Xiong, Chenyan Xiong, Ye Li, et al.2020 · 688 citations
  3. KILT: a Benchmark for Knowledge Intensive Language TasksConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Fabio Petroni, Aleksandra Piktus, Angela Fan, et al.2020 · 678 citations
  4. Well-Read Students Learn Better: On the Importance of Pre-training Compact ModelsarXiv:1908.08962Iulia Turc, Ming-Wei Chang, Kenton Lee, et al.2019 · 678 citations
  5. Improving Multi-hop Question Answering over Knowledge Graphs using Knowledge Base EmbeddingsAnnual Meeting of the Association for Computational Linguistics (ACL)Apoorv Saxena, Aditay Tripathi, Partha Talukdar2020 · 671 citations
  6. CCNet: Extracting High Quality Monolingual Datasets from Web Crawl DataLanguage Resources and Evaluation Conference (LREC)Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, et al.2020 · 666 citations
  7. K-Adapter: Infusing Knowledge into Pre-Trained Models with AdaptersarXiv:2002.01808Ruize Wang, Duyu Tang, Nan Duan, et al.2020 · 666 citations
  8. ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation TransferAnnual Meeting of the Association for Computational Linguistics (ACL)Yuanmeng Yan, Rumei Li, Sirui Wang, et al.2021 · 647 citations
  9. Efficient Intent Detection with Dual Sentence EncodersWorkshop on Natural Language Processing for Conversational AIIñigo Casanueva, Tadas Temčinas, Daniela Gerz, et al.2020 · 628 citations
  10. Towards a Unified View of Parameter-Efficient Transfer LearningarXiv:2110.04366Junxian He, Chunting Zhou, Xuezhe Ma, et al.2021 · 619 citations
  11. Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language ModelarXiv:2201.11990Shaden Smith, Mostofa Patwary, Brandon Norick, et al.2022 · 615 citations
  12. Text Embeddings by Weakly-Supervised Contrastive Pre-trainingarXiv:2212.03533Liang Wang, Nan Yang, Xiaolong Huang, et al.2022 · 598 citations
  13. A Robustly Optimized BERT Pre-training Approach with Post-trainingChinese National Conference on Computational LinguisticsLiu Zhuang, Lin Wayne, Shi Ya, et al.2021 · 587 citations
  14. Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text ModelsFindings of the Association for Computational LinguisticsJianmo Ni, Gustavo Hernandez Abrego, Noah Constant, et al.2021 · 583 citations
  15. AudioLM: a Language Modeling Approach to Audio GenerationIEEE/ACM Transactions on Audio Speech and Language ProcessingZalan Borsos, Raphael Marinier, Damien Vincent, et al.2022 · 581 citations
  16. Learning How to Ask: Querying LMs with Mixtures of Soft PromptsConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Guanghui Qin, Jason Eisner2021 · 573 citations
  17. PhoBERT: Pre-trained language models for VietnameseFindings of the Association for Computational LinguisticsDat Quoc Nguyen, Anh Tuan Nguyen2020 · 571 citations
  18. Extractive Summarization as Text MatchingAnnual Meeting of the Association for Computational Linguistics (ACL)Ming Zhong, Pengfei Liu, Yiran Chen, et al.2020 · 570 citations
  19. What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical ExamsarXiv:2009.13081Di Jin, Eileen Pan, Nassim Oufattole, et al.2020 · 567 citations
  20. MetaMath: Bootstrap Your Own Mathematical Questions for Large Language ModelsarXiv:2309.12284Longhui Yu, Weisen Jiang, Han Shi, et al.2023 · 565 citations
  21. Leveraging Pre-trained Checkpoints for Sequence Generation TasksTransactions of the Association for Computational Linguistics (TACL)Sascha Rothe, Shashi Narayan, Aliaksei Severyn2019 · 564 citations
  22. Reducing Transformer Depth on Demand with Structured DropoutarXiv:1909.11556Angela Fan, Edouard Grave, Armand Joulin2019 · 560 citations
  23. How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and DetectionarXiv:2301.07597Biyang Guo, Xin Zhang, Ziyuan Wang, et al.2023 · 555 citations
  24. Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-TuningAnnual Meeting of the Association for Computational Linguistics (ACL)Armen Aghajanyan, Sonal Gupta, Luke Zettlemoyer2021 · 542 citations
  25. Evaluating Models' Local Decision Boundaries via Contrast SetsMatt Gardner, Yoav Artzi, Victoria Basmov, et al.2020 · 531 citations
  26. How Does NLP Benefit Legal System: A Summary of Legal Artificial IntelligenceAnnual Meeting of the Association for Computational Linguistics (ACL)Haoxi Zhong, Chaojun Xiao, Cunchao Tu, et al.2020 · 518 citations
  27. DeCLUTR: Deep Contrastive Learning for Unsupervised Textual RepresentationsAnnual Meeting of the Association for Computational Linguistics (ACL)John Giorgi, Osvald Nitski, Bo Wang, et al.2021 · 516 citations
  28. TabFact: A Large-scale Dataset for Table-based Fact VerificationarXiv:1909.02164Wenhu Chen, Hongmin Wang, Jianshu Chen, et al.2019 · 515 citations
  29. FinQA: A Dataset of Numerical Reasoning over Financial DataConference on Empirical Methods in Natural Language Processing (EMNLP)Zhiyu Chen, Wenhu Chen, Charese Smiley, et al.2021 · 513 citations
  30. A-OKVQA: A Benchmark for Visual Question Answering using World KnowledgeLecture notes in computer scienceDustin Schwenk, Apoorv Khandelwal, Christopher Clark, et al.2022 · 509 citations
  31. Few-shot Learning with Multilingual Language ModelsarXiv:2112.10668Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, et al.2021 · 500 citations
  32. BBQ: A hand-built bias benchmark for question answeringFindings of the Association for Computational LinguisticsAlicia Parrish, Angelica Chen, Nikita Nangia, et al.2022 · 497 citations
  33. GLaM: Efficient Scaling of Language Models with Mixture-of-ExpertsarXiv:2112.06905Nan Du, Yanping Huang, Andrew M. Dai, et al.2021 · 496 citations
  34. DeeBERT: Dynamic Early Exiting for Accelerating BERT InferenceAnnual Meeting of the Association for Computational Linguistics (ACL)Ji Xin, Raphael Tang, Jaejun Lee, et al.2020 · 487 citations
  35. DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability CurvaturearXiv:2301.11305Eric Mitchell, Yoonho Lee, Alexander Khazatsky, et al.2023 · 481 citations
  36. SimVLM: Simple Visual Language Model Pretraining with Weak SupervisionarXiv:2108.10904Zirui Wang, Jiahui Yu, Adams Wei Yu, et al.2021 · 481 citations
  37. HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingConference on Empirical Methods in Natural Language Processing (EMNLP)Linjie Li, Yen-Chun Chen, Yu Cheng, et al.2020 · 481 citations
  38. ARBERT & MARBERT: Deep Bidirectional Transformers for ArabicarXiv:2101.01785 · ACL-2021 camera ready versionMuhammad Abdul-Mageed, AbdelRahim Elmadany, El Moatez Billah Nagoudi2021 · 478 citations
  39. Data Augmentation using Pre-trained Transformer ModelsarXiv:2003.02245Varun Kumar, Ashutosh Choudhary, Eunah Cho2020 · 474 citations
  40. PPT: Pre-trained Prompt Tuning for Few-shot LearningAnnual Meeting of the Association for Computational Linguistics (ACL)Yuxian Gu, Xu Han, Zhiyuan Liu, et al.2022 · 471 citations
  41. P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and TasksarXiv:2110.07602Xiao Liu, Kaixuan Ji, Yicheng Fu, et al.2021 · 469 citations
  42. OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning FrameworkarXiv:2202.03052Peng Wang, An Yang, Rui Men, et al.2022 · 467 citations
  43. From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual TransformersConference on Empirical Methods in Natural Language Processing (EMNLP)Anne Lauscher, Vinit Ravishankar, Ivan Vulić, et al.2020 · 463 citations
  44. Are All Languages Created Equal in Multilingual BERT?Workshop on Representation Learning for NLPShijie Wu, Mark Dredze2020 · 460 citations
  45. LinkBERT: Pretraining Language Models with Document LinksAnnual Meeting of the Association for Computational Linguistics (ACL)Michihiro Yasunaga, Jure Leskovec, Percy Liang2022 · 458 citations
  46. Ethical and social risks of harm from Language ModelsarXiv:2112.04359Laura Weidinger, John Mellor, Maribeth Rauh, et al.2021 · 456 citations
  47. LLM-Pruner: On the Structural Pruning of Large Language ModelsConference on Neural Information Processing Systems (NeurIPS)Xinyin Ma, Gongfan Fang, Xinchao Wang2023 · 452 citations
  48. Pretrained Transformers Improve Out-of-Distribution RobustnessAnnual Meeting of the Association for Computational Linguistics (ACL)Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, et al.2020 · 452 citations
  49. FreeLB: Enhanced Adversarial Training for Natural Language UnderstandingarXiv:1909.11764Chen Zhu, Yu Cheng, Zhe Gan, et al.2019 · 452 citations
  50. How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language ModelsAnnual Meeting of the Association for Computational Linguistics (ACL)Phillip Rust, Jonas Pfeiffer, Ivan Vulić, et al.2021 · 449 citations
  51. Masked Language Model ScoringAnnual Meeting of the Association for Computational Linguistics (ACL)Julian Salazar, Davis Liang, Toan Q. Nguyen, et al.2020 · 447 citations
  52. Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied AgentsarXiv:2201.07207Wenlong Huang, Pieter Abbeel, Deepak Pathak, et al.2022 · 443 citations
  53. oLMpics — On what Language Model Pre-training CapturesTransactions of the Association for Computational Linguistics (TACL)Alon Talmor, Yanai Elazar, Yoav Goldberg, et al.2019 · 440 citations
  54. Self-Alignment Pretraining for Biomedical Entity RepresentationsConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Fangyu Liu, Ehsan Shareghi, Zaiqiao Meng, et al.2021 · 437 citations
  55. GPT-NER: Named Entity Recognition via Large Language ModelsFindings of the Association for Computational LinguisticsShuhe Wang, Xiaofei Sun, Xiaoya Li, et al.2025 · 436 citations
  56. InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic TasksarXiv:2312.14238Zhe Chen, Jiannan Wu, Wenhai Wang, et al.2023 · 435 citations
  57. Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot FlyAnnual Meeting of the Association for Computational Linguistics (ACL)Nora Kassner, Hinrich Schütze2020 · 435 citations
  58. Security and Privacy Challenges of Large Language Models: A SurveyACM Computing SurveysBadhan Chandra Das, M. Hadi Amini, Yanzhao Wu2024 · 424 citations
  59. Tulu 3: Pushing Frontiers in Open Language Model Post-TrainingarXiv:2411.15124Nathan Lambert, Jacob Morrison, Valentina Pyatkin, et al.2024 · 423 citations
  60. MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text ClassificationAnnual Meeting of the Association for Computational Linguistics (ACL)Jiaao Chen, Zichao Yang, Diyi Yang2020 · 417 citations
  61. Factual Probing Is [MASK]: Learning vs. Learning to RecallConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Zexuan Zhong, Dan Friedman, Danqi Chen2021 · 415 citations
  62. A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health SupportConference on Empirical Methods in Natural Language Processing (EMNLP)Ashish Sharma, Adam Miner, David Atkins, et al.2020 · 412 citations
  63. Do Prompt-Based Models Really Understand the Meaning of Their Prompts?Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)Albert Webson, Ellie Pavlick2022 · 407 citations
  64. Compressing BERT: Studying the Effects of Weight Pruning on Transfer LearningWorkshop on Representation Learning for NLPMitchell Gordon, Kevin Duh, Nicholas Andrews2020 · 407 citations
  65. BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelsarXiv:2106.10199Elad Ben-Zaken, Shauli Ravfogel, Yoav Goldberg2021 · 406 citations
  66. LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical ReasoningInternational Joint Conference on Artificial IntelligenceJian Liu, Leyang Cui, Hanmeng Liu, et al.2020 · 404 citations
  67. A Survey on Recent Approaches for Natural Language Processing in Low-Resource ScenariosConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Michael A. Hedderich, Lukas Lange, Heike Adel, et al.2021 · 401 citations
  68. LongT5: Efficient Text-To-Text Transformer for Long SequencesFindings of the Association for Computational LinguisticsMandy Guo, Joshua Ainslie, David Uthus, et al.2022 · 399 citations
  69. TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in FinanceAnnual Meeting of the Association for Computational Linguistics (ACL)Fengbin Zhu, Wenqiang Lei, Youcheng Huang, et al.2021 · 397 citations
  70. Lawformer: A Pre-trained Language Model for Chinese Legal Long DocumentsAI OpenChaojun Xiao, Xueyu Hu, Zhiyuan Liu, et al.2021 · 396 citations
  71. Aligning AI With Shared Human ValuesarXiv:2008.02275Dan Hendrycks, Collin Burns, Steven Basart, et al.2020 · 395 citations
  72. UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-TrainingarXiv:2002.12804Hangbo Bao, Li Dong, Furu Wei, et al.2020 · 395 citations
  73. Green AIarXiv:1907.10597Roy Schwartz, Jesse Dodge, Noah A. Smith, et al.2019 · 389 citations
  74. Evaluation of Text Generation: A SurveyarXiv:2006.14799Asli Celikyilmaz, Elizabeth Clark, Jianfeng Gao2020 · 388 citations
  75. ETC: Encoding Long and Structured Inputs in TransformersConference on Empirical Methods in Natural Language Processing (EMNLP)Joshua Ainslie, Santiago Ontañón, Chris Alberti, et al.2020 · 386 citations
  76. German's Next Language ModelInternational Conference on Computational Linguistics (COLING)Branden Chan, Stefan Schweter, Timo Möller2020 · 385 citations
  77. FastBERT: a Self-distilling BERT with Adaptive Inference TimeAnnual Meeting of the Association for Computational Linguistics (ACL)Weijie Liu, Peng Zhou, Zhiruo Wang, et al.2020 · 383 citations
  78. Transformers as Soft Reasoners over LanguageInternational Joint Conference on Artificial IntelligencePeter Clark, Oyvind Tafjord, Kyle Richardson2020 · 382 citations
  79. Splitwise: Efficient generative LLM inference using phase splittingISCA 2024Pratyush Patel, Esha Choukse, Chaojie Zhang, et al.2023 · 379 citations
  80. The GEM Benchmark: Natural Language Generation, its Evaluation and MetricsWorkshop on Generation, Evaluation and Metrics (GEM)Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, et al.2021 · 369 citations
  81. Revisiting Few-sample BERT Fine-tuningarXiv:2006.05987Tianyi Zhang, Felix Wu, Arzoo Katiyar, et al.2020 · 368 citations
  82. Parameter-Efficient Transfer Learning with Diff PruningAnnual Meeting of the Association for Computational Linguistics (ACL)Demi Guo, Alexander Rush, Yoon Kim2021 · 367 citations
  83. The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language ModelsarXiv:2103.06678Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, et al.2021 · 365 citations
  84. Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text ClassificationAnnual Meeting of the Association for Computational Linguistics (ACL)Shengding Hu, Ning Ding, Huadong Wang, et al.2022 · 362 citations
  85. TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented DialogueConference on Empirical Methods in Natural Language Processing (EMNLP)Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher, et al.2020 · 361 citations
  86. BERTje: A Dutch BERT ModelarXiv:1912.09582Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, et al.2019 · 361 citations
  87. BRIO: Bringing Order to Abstractive SummarizationAnnual Meeting of the Association for Computational Linguistics (ACL)Yixin Liu, Pengfei Liu, Dragomir Radev, et al.2022 · 360 citations
  88. Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and InferencearXiv:2412.13663Benjamin Warner, Antoine Chaffin, Benjamin Clavié, et al.2024 · 358 citations
  89. XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and GenerationConference on Empirical Methods in Natural Language Processing (EMNLP)Yaobo Liang, Nan Duan, Yeyun Gong, et al.2020 · 358 citations
  90. Reformulating Unsupervised Style Transfer as Paraphrase GenerationKalpesh Krishna, John Wieting, Mohit Iyyer2020 · 350 citations
  91. Symbolic Knowledge Distillation: from General Language Models to Commonsense ModelsConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Peter West, Chandra Bhagavatula, Jack Hessel, et al.2022 · 349 citations
  92. Time-Aware Language Models as Temporal Knowledge BasesTransactions of the Association for Computational Linguistics (TACL)Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, et al.2022 · 349 citations
  93. RobBERT: a Dutch RoBERTa-based Language ModelarXiv:2001.06286Pieter Delobelle, Thomas Winters, Bettina Berendt2020 · 349 citations
  94. UL2: Unifying Language Learning ParadigmsarXiv:2205.05131Yi Tay, Mostafa Dehghani, Vinh Q. Tran, et al.2022 · 347 citations
  95. Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary SpaceConference on Empirical Methods in Natural Language Processing (EMNLP)Mor Geva, Avi Caciularu, Kevin Ro Wang, et al.2022 · 346 citations
  96. Directed Acyclic Graph Network for Conversational Emotion RecognitionAnnual Meeting of the Association for Computational Linguistics (ACL)Weizhou Shen, Siyue Wu, Yunyi Yang, et al.2021 · 346 citations
  97. SimCLS: A Simple Framework for Contrastive Learning of Abstractive SummarizationAnnual Meeting of the Association for Computational Linguistics (ACL)Yixin Liu, Pengfei Liu2021 · 341 citations
  98. WARP: Word-level Adversarial ReProgrammingarXiv:2101.00121Karen Hambardzumyan, Hrant Khachatrian, Jonathan May2021 · 340 citations
  99. RLPrompt: Optimizing Discrete Text Prompts with Reinforcement LearningConference on Empirical Methods in Natural Language Processing (EMNLP)Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, et al.2022 · 338 citations
  100. Neural Codec Language Models are Zero-Shot Text to Speech SynthesizersarXiv:2301.02111Chengyi Wang, Sanyuan Chen, Yu Wu, et al.2023 · 335 citations

1 2 3 … 10 … 15

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/cited-by/page/2/