Elamur Research
☰
home / papers / 2019 / Release Strategies and the Social Impacts of Language Models

Release Strategies and the Social Impacts of Language Models

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1908.09203

Summary

This OpenAI report covers how GPT-2, a large unsupervised text generator, was released and what the team learned about its effects on society. GPT-2 was not published all at once. OpenAI put out four model sizes over about nine months, so it could study risks and benefits before releasing each larger version. Alongside the releases, it ran research partnerships with outside institutions on misuse, detection, human perception and bias. It also published a dataset of model outputs and contributed to Partnership on AI discussions on publication norms. The authors found little evidence of planned misuse and many beneficial applications, which gave them enough confidence to release the full 1.5B-parameter model. They also expect the skill and cost needed to use such models to keep falling. The report matters because it was an early, concrete example of responsible-release practice. Its proposals for the AI community cover decision frameworks, shared risk analysis and better communication between labs.

Key points

  • Release timeline: 124M parameters (February 2019), 355M (May), 774M (August) and 1.5B (November).
  • The May release also included a dataset of samples from all four model sizes, meant for training detectors (both human and automatic) and for studying bias.
  • The four partners were Cornell University (how easily people are taken in by synthetic disinformation), Middlebury's CTEC (use by extremists), the University of Oregon (bias probes) and UT Austin (detecting fine-tuned outputs and whether detection transfers between models). A generic model-sharing agreement is in Appendix A.
  • Cornell found that, for human readers, larger models gave smaller and smaller gains in how hard their text was to detect. Other labs also released cautiously, for example GROVER (AI2/UW), Salesforce's CTRL and Hugging Face.
  • There are three recommendations: build frameworks for weighing tradeoffs, build infrastructure for distributed risk analysis (with attention to scalability, security and fairness), and set up communication channels between organizations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

263 papers of our library cite it (of the 140673 whose bibliographies we read); 15 of them have a page here, 248 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1908.09203 · arXiv 1908.09203

@misc{solaiman2019release,
  title = {{Release Strategies and the Social Impacts of Language Models}},
  author = {Solaiman, Irene and Brundage, Miles and Clark, Jack and Askell, Amanda and Herbert-Voss, Ariel and Wu, Jeff and Radford, Alec and Krueger, Gretchen and Kim, Jong Wook and Kreps, Sarah and McCain, Miles and Newhouse, Alex and Blazakis, Jason and McGuffie, Kris and Wang, Jasmine},
  year = {2019},
  doi = {10.48550/arxiv.1908.09203},
  eprint = {1908.09203},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1908.09203}
}

References

81 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Ifeoluwa Adelani, Haotian Mai, Fuming Fang, Huy H. Nguyen, Junichi Yamagishi, and Isao Echizen. Generating sentiment-preserving fake online reviews using neural language models and their human- and machine-based detection. arXiv preprint arXiv:1907.09177, 2019. — [arXiv:1907.09177](https://arxiv.org/abs/1907.09177) find
  2. Sakshi Agarwal and Lav R. Varshney. Limits of deepfake detection: A robust estimation viewpoint, 2019. find
  3. Dimitrios Alikaniotis and Vipul Raheja. The unreasonable effectiveness of transformer language models in grammatical error correction. arXiv preprint arXiv:1906.01733, 2019. find
  4. Anonymous. Plug and play language model: A simple baseline for controlled language generation. In Submitted to International Conference on Learning Representations, 2020. URL ?iiTb,ff QT2M\`2pB2rXM2if7Q\`mK\\B/4\>R2/1v"E.a. under review. find
  5. Anonymous. Reducing sentiment bias in language models via counterfactual evaluation. In Submitted to International Conference on Learning Representations, 2020. URL ?iiTb,ffQT2M\`2pB2rX M2if7Q\`mK\\B/4aRHkAv\`uS\`. under review. find
  6. Anton Bakhtin, Sam Gross, Myle Ott, Yuntian Deng, Marc’Aurelio Ranzato, and Arthur Szlam. Real or fake? learning to discriminate machine from human generated text. arXiv preprint arXiv:1906.03351, 2019. — [arXiv:1906.03351](https://arxiv.org/abs/1906.03351) find
  7. Iz Beltagy, Arman Cohan, and Kyle Lo. SciBERT: Pretrained Contextualized Embeddings for Scientific Text. arXiv preprint arXiv:1903.10676, 2019. find our card
  8. Emily M. Bender and Batya Friedman. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604, 2018. doi: 10.1162/tacl\_a\_00041. URL ?iiTb,ffrrrX+Hr2\#X Q\`;fMi?QHQ;vfZR3@Ry9R. find
  9. Jayadev Bhaskaran and Isha Bhallamudi. Good secretaries, bad truck drivers? Occupational gender stereotypes in sentiment analysis. arXiv preprint arXiv:1906.10256, 2019. find
  10. Siddharth Biswal, Cao Xiao, M. Brandon Westover, and Jimeng Sun. EEGtoText: Learning to write medical reports from EEG recordings. In Proceedings of Machine Learning Research, volume 106 of Proceedings of Machine Learning Research, pages 1–18. PMLR, 2019. find
  11. Gwern Branwen. GPT-2 Neural Network Poetry. Mar 2019. URL ?iiTb,ffrrrX;r2\`MXM2if :Sh@k. (Accessed on 08/15/2019). find
  12. Paweł Budzianowski and Ivan Vulić. Hello, it’s gpt-2 – how can i help you? towards the use of pretrained language models for task-oriented dialogue systems. arXiv preprint arXiv:1907.05774, 2019. find
  13. Yaroslav Bulatov. Large-scale ai and sharing of models. Jul 2019. URL ?iiTb,ffK2/BmKX+QKf !v\`QbHpp\#fH\`;2@b+H2@B@M/@b?\`BM;@Q7@KQ/2Hb@9ekk\#8N2+R3. (Accessed on 08/19/2019). Quickfacts united states: Race and hispanic origin. find
  14. U.S. Census Bureau. ffrrrX+2MbmbX;Qpf[mB+F7+ibf7+ifi\#H2flafSahy98kR3OSahy98kR3. 08/19/2019). find
  15. Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186, Apr 2017. ISSN 0036-8075. doi: 10.1126/science.aal4230. — [doi:10.1126/science.aal4230](https://doi.org/10.1126/science.aal4230) find
  16. Pew Research Center. Global religious diversity. Apr 2014. URL ?iiTb,ffrrrXT2r7Q\`mKXQ\`;f kyR9fy9fy9f;HQ\#H@\`2HB;BQmb@/Bp2\`bBivf. (Accessed on 08/15/2019). find
  17. Andrew M. Dai and Quoc V. Le. arXiv:1511.01432, 2015. — [arXiv:1511.01432](https://arxiv.org/abs/1511.01432) find
  18. Samina Dazdarevic, Ana Stišović Milovanović, and Fahreta Fijuljanin. Translating sacred words. In 5th International Social Sciences Conference, Jun 2013. find
  19. Clément Delangue. Ethical analysis of the open-sourcing of a state-of-the-art conversational AI. May 2019. URL ?iiTb,ffK2/BmKX+QKf?m;;BM;7+2f2i?B+H@MHvbBb@Q7@i?2@QT2M@ bQm\`+BM;@Q7@@bii2@Q7@i?2@\`i@+QMp2\`biBQMH@B@38kRRj+jk9\#k. (Accessed on 08/15/2019). find
  20. Hugging Face. Write with transformer. 2019. URL ?iiTb,ffi\`Mb7Q\`K2\`X?m;;BM;7+2X+Qf. (Accessed on 08/15/2019). find
  21. Hugging Face. Swift core ml implementations of transformers. 2019. URL ?iiTb,ff;Bi?m\#X +QKf?m;;BM;7+2fbrB7i@+Q\`2KH@i\`Mb7Q\`K2\`b. (Accessed on 08/15/2019). find
  22. FBI. Table 43: Arrests by race and ethnicity, 2017. URL ?iiTb,ffm+\`X7\#BX;Qpf+\`BK2@BM@ i?2@mXbfkyRdf+\`BK2@BM@i?2@mXbX@kyRdfi\#H2bfi\#H2@9j. (Accessed on 08/19/2019). find
  23. Xavier Ferrer, Jose Such, and Natalia Criado. Attesting biases and discrimination using language semantics. In Responsible Artificial Intelligence Agents WS of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS’19), Apr 2019. URL ?iiTb, (Accessed on Semi-supervised seque… find
  24. Jonathan Fly. Testing the limits of Grover the neural fake news detector. Can it write fiction? Can it write riddles? May 2019. URL ?iiTb,ffB7Q\`+2/\#QiX+QKfr?i@+M@@7F2@M2rb@ /2i2+iQ\`@/Qf. (Accessed on 08/15/2019). find
  25. Allen Institute for Artificial Intelligence. GPT-2 explorer. 2019. URL ?iiTb,ff;TikXTTbX HH2MBXQ\`;f\\i2ti4CQ2HWkyBb. (Accessed on 08/19/2019). find
  26. Centers for Disease Control and Prevention. survey (NISVS) infographic. Apr 2017. URL ?iiTb,ffrrrX+/+X;QpfpBQH2M+2T\`2p2MiBQMf +QKKmMB+iBQM\`2bQm\`+2bfBM7Q;\`T?B+bfBM7Q;\`T?B+X?iKH\\*.*nn\`27oH. cessed on 08/15/2019). find
  27. Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. GLTR: Statistical detection and visualization of generated text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 111–116, Florence, Italy, July 2019. Association for Com… find our card
  28. Aaron Gokaslan and Vanya Cohen. URL ?iiTb,ff\#HQ;Xmb2DQm\`MHX+QKfQT2M;Ti@k@r2@\`2THB+i2/@;Ti@k@ Aug 2019. \#2+mb2@vQm@+M@iQQ@982j92e/je/+. (Accessed on 11/04/2019). find
  29. Matthew Groh, Ziv Epstein, Nick Obradovich, Manuel Cebrian, and Iyad Rahwan. Human detection of machine manipulated media. arXiv preprint arXiv:1907.05276, 2019. — [arXiv:1907.05276](https://arxiv.org/abs/1907.05276) find
  30. Jiaqi Guan, Runzhe Li, Sheng Yu, and Xuegong Zhang. Generation of synthetic electronic medical record text. In IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2018, Madrid, Spain, December 3-6, 2018, pages 374–380, 2018. doi: 10.1109/BIBM.2018.8621223. URL ?iiT,ff/QBXB222+QKTmi… — [doi:10.1109/bibm.2018.8621223](https://doi.org/10.1109/bibm.2018.8621223) find
  31. Santosh Gupta. Docproduct: Medical Q&A with deep language models. 2019. URL ?iiTb, ff;Bi?m\#X+QKf\`2@b2\`+?f.Q+S\`Q/m+i. (Accessed on 08/15/2019). find
  32. Perry Hinton. Implicit stereotypes and the predictive brain: cognition and culture in “biased” person perception. Palgrave Communications, 3:17086, 2017. find
  33. Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751, 2019. find
  34. Dirk Hovy and Shannon L Spruit. The social impact of natural language processing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 591–598, 2016. National intimate partner and sexual violence (Ac-Opengpt-2: We replicated gpt-2… find our card
  35. Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018. doi: 10.18653/v1/p18-1031. URL ?iiT,ff/tX/QBXQ\`;fRyXR3e8jfpRf TR3@RyjR. find
  36. This Week in Machine Learning & AI. Dissecting the controversy surrounding OpenAI’s new language model. Feb 2019. URL ?iiTb,ffirBKHBX+QKfirBKH@iHF@kj9@/Bbb2+iBM;@i?2@ +QMi\`Qp2\`bv@bm\`\`QmM/BM;@QT2MBb@M2r@HM;m;2@KQ/2Hf. (Accessed on 08/15/2019). find
  37. Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. Human and automatic detection of generated text, 2019. find
  38. Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation, 2019. find our card
  39. Adam King. Talk to transformer. URL ?iiTb,ffiHFiQi\`Mb7Q\`K2\`X+QKf. (Accessed on 08/15/2019). find
  40. Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. Measuring bias in contextualized word representations, 2019. find our card
  41. AI21 Labs. Haim: A modest step towards controllable text generation. URL ?iiTb,ffrrrXBkRX +QKf?BK@TQbi. find
  42. Connor Leahy. Replicating gpt-2 1.5b. Jun 2019. URL ?iiTb,ffK2/BmKX+QKf!LS*QHHTb2f \`2THB+iBM;@;Tik@R@8\#@3e989d7ke7. (Accessed on 11/04/2019). find
  43. Connor Leahy. The hacker learns to trust. Jun 2019. URL ?iiTb,ffK2/BmKX+QKf!LS*QHHTb2f i?2@?+F2\`@H2\`Mb@iQ@i\`mbi@ek7j+R9Ny78R. (Accessed on 11/04/2019). find
  44. Peter Lee. Learning from Tay’s introduction. The Official Microsoft Blog, Mar 2016. URL ?iiTb, ff\#HQ;bXKB+\`QbQ7iX+QKf\#HQ;fkyRefyjfk8fH2\`MBM;@ivb@BMi\`Q/m+iBQMf. (Accessed on 08/15/2019). find
  45. Guanxiong Liu, Tzu-Ming Harry Hsu, Matthew McDermott, Willie Boag, Wei-Hung Weng, Peter Szolovits, and Marzyeh Ghassemi. Clinically accurate chest x-ray report generation. arXiv preprint arXiv:1904.02633, 2019. — [arXiv:1904.02633](https://arxiv.org/abs/1904.02633) find
  46. Microsoft. Microsoft anti-spam policy: Office 2007. +QKf2M@mbf\`iB+H2fKB+\`QbQ7i@MiB@bTK@TQHB+v@298ye7Nd@eN97@9N\#+@3kjR@ ++9jeN7+\#3. (Accessed on 08/19/2019). URL ?iiTb,ffbmTTQ\`iXQ77B+2X find
  47. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency - FAT* ’19, 2019. doi: 10.1145/3287560.32… find
  48. Rachel E. Morgan and Jennifer L. Truman. Criminal victimization, 2017. Bureau of Justice Statistics, 251150, Dec 2018. URL ?iiTb,ffrrrX\#DbX;Qpf+QMi2MifTm\#fT/7f+pRdXT/7. find
  49. Adam Mosseri. Working to stop misinformation and false news. URL ?iiTb,ffrrrX7+2\#QQFX+QKf7+2\#QQFK2/Bf\#HQ;frQ\`FBM;@iQ@biQT@ Apr 2017. KBbBM7Q\`KiBQM@M/@7Hb2@M2rb. (Accessed on 08/19/2019). find
  50. Malvina Nissim, Rik van Noord, and Rob van der Goot. Fair is better than sensational:man is to doctor as woman is to doctor. arXiv preprint arXiv:1905.09866, 2019. find
  51. Bureau of Justice Statistics. Data collection: National crime victimization survey (ncvs). 1973-2017. URL ?iiTb,ffrrrX\#DbX;QpfBM/2tX+7K\\iv4/+/2iBHBB/4k98. 08/15/2019). find
  52. UNC School of Medicine Psychiatric Genomics Consortium. 2019. URL ?iiTb,ffrrrXK2/XmM+X2/mfT;+fb?\`2/@K2i?Q/bf?Qr@iQf. 08/19/2019). find
  53. OJJDP. Arrests by offense, age, and gender: 2017. QDbii\#\#f+\`BK2fm+\`XbT\\i\#H2nBM4Rb2Hu\`b4kyRd\`/Q:\`QmTb4k\`/Q.i4+. cessed on 08/19/2019). find
  54. Oluwatobi Olabiyi and Erik T Mueller. Multi-turn dialogue response generation with autoregressive transformer models. arXiv preprint arXiv:1908.01841, 2019. — [arXiv:1908.01841](https://arxiv.org/abs/1908.01841) find
  55. OpenAI. Better language models and their implications. OpenAI Blog, Feb 2019. URL ?iiTb, ffQT2MBX+QKf\#HQ;f\#2ii2\`@HM;m;2@KQ/2Hbf. (Accessed on 08/15/2019). find
  56. OpenAI. GPT-2 output dataset. 2019. URL ?iiTb,ff;Bi?m\#X+QKfQT2MBf;Ti@kf\#HQ\#f Kbi2\`f/QKBMbXiti. (Accessed on 11/1/2019). find
  57. OpenAI. GPT-2 detector model. 2019. URL ?iiTb,ff;Bi?m\#X+QKfQT2MBf;Ti@k@QmiTmi@ /ib2ifi\`22fKbi2\`f/2i2+iQ\`. (Accessed on 11/1/2019). find
  58. OpenAI. GPT-2. 2019. URL ?iiTb,ff;Bi?m\#X+QKfQT2MBf;Ti@k. (Accessed on 08/15/2019). find
  59. OpenAI. GPT-2 model card. 2019. URL ?iiTb,ff;Bi?m\#X+QKfQT2MBf;Ti@kf\#HQ\#fKbi2\`f KQ/2Hn+\`/XK/. (Accessed on 11/1/2019). Facebook for Media, (Accessed on How to request data access. (Accessed on URL ?iiTb,ffrrrXQDD/TX;Qpf (Ac- find
  60. OpenAI. MuseNet. OpenAI Blog, Apr 2019. URL ?iiTb,ffQT2MBX+QKf\#HQ;fKmb2M2if. (Accessed on 08/19/2019). find
  61. Aviv Ovadya and Jess Whittlestone. Reducing malicious use of synthetic media research: Considerations and potential release practices for machine learning. CoRR, abs/1907.11274, 2019. URL ?iiT,ff\`tBpXQ\`;f\#bfRNydXRRkd9. — [arXiv:1907.11274](https://arxiv.org/abs/1907.11274) find
  62. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volu… find
  63. Alec Radford. Language models and their uses. Apr 2019. URL ?iiTb,ffrrrXvQmim\#2X+QKf ri+?\\p4:1i\#.eT[hh1. (Accessed on 08/19/2019). find
  64. Alec Radford, Jeffrey Wu, et al. Language models are unsupervised multitask learners. 2019. find
  65. Yoel Roth and Del Harvey. How Twitter is fighting spam and malicious automation. URL ?iiTb,ff\#HQ;XirBii2\`X+QKf2MnmbfiQTB+bf+QKTMvf Twitter Blog, Jun 2018. kyR3f?Qr@irBii2\`@Bb@7B;?iBM;@bTK@M/@KHB+BQmb@miQKiBQMX?iKH. on 08/19/2019). find
  66. Tal Schuster, Roei Schuster, Darsh J Shah, and Regina Barzilay. Are we safe yet? the limitations of distributional features for fake news detection, 2019. find
  67. Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. What makes a good conversation? How controllable attributes affect human judgments. arXiv preprint arXiv:1902.08654, 2019. find our card
  68. Janelle Shane. GPT-2: It learned on the internet. Feb 2019. URL ?iiTb,ffBr2B\`/M2bbX+QKf TQbifR3k3k9dR8k8df;Ti@k@Bi@H2\`M2/@QM@i?2@BMi2\`M2i. (Accessed on 08/15/2019). find
  69. Ilya Sutskever. GPT-2. Apr 2019. URL ?iiTb,ffrrrXvQmim\#2X+QKfri+?\\p4hyA33L?\_nNJ. (Accessed on 08/15/2019). find
  70. TabNine. Autocompletion with deep learning. Jul 2019. URL ?iiTb,ffi\#MBM2X+QKf\#HQ;f /22T. (Accessed on 08/15/2019). find
  71. UNODC. Global study on homocide: Gender-related killing of women and girls. Nov 2018. find
  72. Lav R. Varshney, Nitish Shirish Keskar, and Richard Socher. Pretrained ai models: Performativity, mobility, and change, 2019. (Accessed find
  73. James Vincent. There’s a subreddit populated entirely by AI personifications of other subreddits. The Verge, Jun 2019. URL ?iiTb,ffrrrXi?2p2\`;2X+QKfkyRNfefefR3e88kRkf\`2//Bi@ B@\#Qib@;Tik@QT2MB@i2ti@\`iB7B+BH@BMi2HHB;2M+2@bm\#\`2//Bi. 08/15/2019). find
  74. Nick Walton. AI Dungeon. URL ?iiT,ffB/mM;2QMXBQf. (Accessed on 08/15/2019). find
  75. Jess Whittlestone, Rune Nyrup, Anna Alexandrova, and Stephen Cave. The role and limits of principles in AI ethics: Towards a focus on tensions. In Proceedings of the AAAI/ACM Conference on AI Ethics and Society, Honolulu, HI, USA, pages 27–28, 2019. find
  76. Thomas Wolf. How to build a state-of-the-art conversational AI with transfer learning. May 2019. URL ?iiTb,ffK2/BmKX+QKf?m;;BM;7+2f?Qr@iQ@\#mBH/@@bii2@Q7@ i?2@\`i@+QMp2\`biBQMH@B@rBi?@i\`Mb72\`@H2\`MBM;@k/3R3+kejRj. (Accessed on 08/15/2019). find
  77. Max Woolf. Experiments with making convincing ai-generated fake news. Sep 2019. URL ?iiTb, ffKBMBKtB\`X+QKfkyRNfyNf+i\`H@7F2@M2rbf. (Accessed on 11/08/2019). find
  78. Kai-Cheng Yang, Onur Varol, Clayton A. Davis, Emilio Ferrara, Alessandro Flammini, and Filippo Menczer. Arming the public with artificial intelligence to counter social bots. Human Behavior and Emerging Technologies, 1(1):48–61, 2019. doi: 10.1002/hbe2.115. URL ?iiTb, ffQMHBM2HB\#\`\`vXrBH2vX+QKf/QBf… — [doi:10.1002/hbe2.115](https://doi.org/10.1002/hbe2.115) find
  79. Rowan Zellers. Why we released Grover. The Gradient, Jul 2019. URL ?iiTb,ffi?2;\`/B2MiX Tm\#fr?v@r2@\`2H2b2/@;\`Qp2\`f. (Accessed on 08/15/2019). find
  80. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. arXiv preprint arXiv:1905.12616, 2019. find our card
  81. Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. Ernie: Enhanced language representation with informative entities. arXiv preprint arXiv:1905.07129, 2019. (Accessed on find