PaLM 2 Technical Report

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

812 papers of our library cite it (of the 140673 whose bibliographies we read); 80 of them have a page here, 732 not yet.

• Large Language Models for Mathematical Reasoning: Progresses and Challenges
• The Falcon Series of Open Language Models
• Tower: An Open Multilingual Large Language Model for Translation-Related Tasks
• Aya 23: Open Weight Releases to Further Multilingual Progress
• Qwen Technical Report
• Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
• AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors
• AlpaGasus: Training A Better Alpaca with Fewer Data
• AlphaMath Almost Zero: Process Supervision without Process
• Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models
• Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• SeamlessM4T: Massively Multilingual & Multimodal Machine Translation
• Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models
• Safe RLHF: Safe Reinforcement Learning from Human Feedback
• The Devil Is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation
• Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution
• Results of WMT23 Metrics Shared Task: Metrics Might Be Guilty but References Are Not Innocent
• Gemini: A Family of Highly Capable Multimodal Models
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
• ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving
• MiniLLM: On-Policy Distillation of Large Language Models
• Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation
• Large Language Models Cannot Self-Correct Reasoning Yet
• TrustLLM: Trustworthiness in Large Language Models
• Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
• Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2
• Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity
• BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset
• MetricX-23: The Google Submission to the WMT 2023 Metrics Shared Task
• Challenges and Applications of Large Language Models
• DistiLLM: Towards Streamlined Distillation for Large Language Models
• HunyuanVideo: A Systematic Framework For Large Video Generative Models
• MADLAD-400: A Multilingual And Document-Level Large Audited Dataset
• RA-DIT: Retrieval-Augmented Dual Instruction Tuning
• Video-LLaVA: Learning United Visual Representation by Alignment Before Projection
• AgentBench: Evaluating LLMs as Agents
• DADA: Dialect Adaptation via Dynamic Aggregation of Linguistic Rules
• WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct
• GAIA: a benchmark for General AI Assistants
• Orca 2: Teaching Small Language Models How to Reason
• Generative Representational Instruction Tuning
• BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages
• A Comprehensive Overview of Large Language Models
• Iterative Reasoning Preference Optimization
• Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions
• Code Llama: Open Foundation Models for Code
• AudioPaLM: A Large Language Model That Can Speak and Listen
• "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models
• Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
• Aligning Large Multimodal Models with Factually Augmented RLHF
• SALMONN: Towards Generic Hearing Abilities for Large Language Models
• Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models
• Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
• Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model
• How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources
• MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning
• PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization
• Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations
• Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
• Long-form factuality in large language models
• Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts
• Benchmarking Retrieval-Augmented Generation for Medicine
• Corrective Retrieval Augmented Generation
• Baichuan 2: Open Large-scale Language Models
• The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)
• Large Language Models as Optimizers
• Rethinking Benchmark and Contamination for Language Models with Rephrased Samples
• GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts
• MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models
• MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
• Scaling Relationship on Learning Mathematical Reasoning with Large Language Models
• MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning
• TinyLlama: An Open-Source Small Language Model
• A Survey of Large Language Models
• LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset
• Instruction-Following Evaluation for Large Language Models
• Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification
• WebArena: A Realistic Web Environment for Building Autonomous Agents

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.10403 · arXiv 2305.10403

@misc{anil2023palm,
  title = {{PaLM 2 Technical Report}},
  author = {Anil, Rohan and Dai, Andrew M. and Firat, Orhan and Johnson, Melvin and Lepikhin, Dmitry and Passos, Alexandre and Shakeri, Siamak and Taropa, Emanuel and Bailey, Paige and Chen, Zhifeng and Chu, Eric and Clark, Jonathan H. and Shafey, Laurent El and Huang, Yanping and Meier-Hellstern, Kathy and Mishra, Gaurav and Moreira, Erica and Omernick, Mark and Robinson, Kevin and Ruder, Sebastian and Tay, Yi and Xiao, Kefan and Xu, Yuanzhong and Zhang, Yujing and Abrego, Gustavo Hernandez and Ahn, Junwhan and Austin, Jacob and Barham, Paul and Botha, Jan and Bradbury, James and Brahma, Siddhartha and Brooks, Kevin and Catasta, Michele and Cheng, Yong and Cherry, Colin and Choquette-Choo, Christopher A. and Chowdhery, Aakanksha and Crepy, Clement and Dave, Shachi and Dehghani, Mostafa and Dev, Sunipa and Devlin, Jacob and Diaz, Mark and Du, Nan and Dyer, Ethan and Feinberg, Vlad and Feng, Fangxiaoyu and Fienber, Vlad and Freitag, Markus and Garcia, Xavier and Gehrmann, Sebastian and Gonzalez, Lucas and Gur-Ari, Guy and Hand, Steven and Hashemi, Hadi and Hou, Le and Howland, Joshua and Hu, Andrea and Hui, Jeffrey and Hurwitz, Jeremy and Isard, Michael and Ittycheriah, Abe and Jagielski, Matthew and Jia, Wenhao and Kenealy, Kathleen and Krikun, Maxim and Kudugunta, Sneha and Lan, Chang and Lee, Katherine and Lee, Benjamin and Li, Eric and Li, Music and Li, Wei and Li, YaGuang and Li, Jian and Lim, Hyeontaek and Lin, Hanzhao and Liu, Zhongtao and Liu, Frederick and Maggioni, Marcello and Mahendru, Aroma and Maynez, Joshua and Misra, Vedant and Moussalem, Maysam and Nado, Zachary and Nham, John and Ni, Eric and Nystrom, Andrew and Parrish, Alicia and Pellat, Marie and Polacek, Martin and Polozov, Alex and Pope, Reiner and Qiao, Siyuan and Reif, Emily and Richter, Bryan and Riley, Parker and Ros, Alex Castro and Roy, Aurko and Saeta, Brennan and Samuel, Rajkumar and Shelby, Renee and Slone, Ambrose and Smilkov, Daniel and So, David R. and Sohn, Daniel and Tokumine, Simon and Valter, Dasha and Vasudevan, Vijay and Vodrahalli, Kiran and Wang, Xuezhi and Wang, Pidong and Wang, Zirui and Wang, Tao and Wieting, John and Wu, Yuhuai and Xu, Kelvin and Xu, Yunhan and Xue, Linting and Yin, Pengcheng and Yu, Jiahui and Zhang, Qiao and Zheng, Steven and Zheng, Ce and Zhou, Weikang and Zhou, Denny and Petrov, Slav and Wu, Yonghui},
  year = {2023},
  doi = {10.48550/arxiv.2305.10403},
  eprint = {2305.10403},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.10403}
}

References

159 references, 44 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abid, A., Farooqi, M., and Zou, J. Persistent anti-muslim bias in large language models. arXiv:2101.05783, 2021. URL https://arxiv.org/abs/2101.05783. arXiv preprint Akhbardeh, F., Arkhangorodsky, A., Biesialska, M., Bojar, O., Chatterjee, R., Chaudhary, V., Costa-jussa, M. R., España-Bonet, C., Fan… ↗ our card
2. Appen. Guide to fair pay, 2023. 9557008940941-Guide-to-Fair-Pay. find
3. URL https://success.appen.com/hc/en-us/articles/Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. URL https: //arxiv.org/abs/2108.07732. ↗ our card
4. Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C.,… find
5. Bapna, A., Caswell, I., Kreutzer, J., Firat, O., van Esch, D., Siddhant, A., Niu, M., Baljekar, P., Garcia, X., Macherey, W., Breiner, T., Axelrod, V., Riesa, J., Cao, Y., Chen, M. X., Macherey, K., Krikun, M., Wang, P., Gutkin, A., Shah, A., Huang, Y., Chen, Z., Wu, Y., and Hughes, M. Building mach… — [arXiv:2205.03983](https://arxiv.org/abs/2205.03983) find
6. Barham, P., Chowdhery, A., Dean, J., Ghemawat, S., Hand, S., Hurt, D., Isard, M., Lim, H., Pang, R., Roy, S., et al. Pathways: Asynchronous distributed dataflow for ml. Proceedings of Machine Learning and Systems, 4:430–449, 2022. find
7. Barocas, S., Hardt, M., and Narayanan, A. Fairness and machine learning limitations and opportunities. 2017. find
8. Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M. R., Vaughan, J. W., Wadsworth, W. D., and Wallach, H. Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’21, pp. 368–378, N… — [doi:10.1145/3461702.3462610](https://doi.org/10.1145/3461702.3462610) find
9. Bender, E. M. and Friedman, B. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604, 2018. doi: 10.1162/tacl\_a\_00041. URL https://aclanthology.org/Q18-1041. ↗
10. Berant, J., Chou, A., Frostig, R., and Liang, P. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1533–1544, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL h… find
11. Bhatt, S., Dev, S., Talukdar, P., Dave, S., and Prabhakaran, V. Re-contextualizing fairness in NLP: The case of india. September 2022. URL https://arxiv.org/abs/2209.12226. ↗
12. Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432–7439, 2020. find our card
13. Blodgett, S. L., Barocas, S., Daumé, III, H., and Wallach, H. Language (technology) is power: A critical survey of “bias” in NLP. May 2020. URL https://arxiv.org/abs/2005.14050. ↗ our card
14. Blodgett, S. L., Lopez, G., Olteanu, A., Sim, R., and Wallach, H. Stereotyping Norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural… find
15. Borkan, D., Dixon, L., Sorensen, J., Thain, N., and Vasserman, L. Nuanced metrics for measuring unintended bias with real data for text classification, 2019. URL https://arxiv.org/abs/1903.04561. — [arXiv:1903.04561](https://arxiv.org/abs/1903.04561) ↗
16. Bowman, S. R. and Dahl, G. E. What will it take to fix benchmarking in natural language understanding?, 2021. find
17. Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax. ↗
18. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess… find
19. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper\_files/paper/2020/file/ 1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf. ↗
20. Carlini, N., Liu, C., Erlingsson, Ú., Kos, J., and Song, D. The secret sharer: Evaluating and testing unintended memorization in neural networks. In USENIX Security Symposium, volume 267, 2019. find our card
21. Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T. B., Song, D., Erlingsson, U., et al. Extracting training data from large language models. In USENIX Security Symposium, volume 6, 2021. find our card
22. Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., and Zhang, C. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646, 2022. URL https://arxiv.org/abs/2202.07646. ↗
23. Casad, B. J., Hale, P., and Wachs, F. L. Stereotype threat among girls: Differences by gender identity and math education context, 2017. find
24. Chen, K., Xu, W., Cheng, X., Xiaochuan, Z., Zhang, Y., Song, L., Wang, T., Qi, Y., and Chu, W. Question directed graph attention network for numerical reasoning over text. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6759–6768, Online, Novemb… find
25. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. URL https://arxiv.org/abs/2107.03374. ↗ our card
26. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., et al. PaLM: Scaling language modeling with Pathways. arXiv preprint arXiv:2204.02311, 2022. URL https://arxiv.org/abs/2204.02311. ↗
27. Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Castro-Ros, A., Pellat, M., Robinson, K., Valter, D., Narang, S., Mishra, G., Yu, A., Zhao, V., Huang, Y., Dai, A., Yu, H… find
28. Clark, J. H., Choi, E., Collins, M., Garrette, D., Kwiatkowski, T., Nikolaev, V., and Palomaki, J. TyDiQA: A benchmark for information-seeking question answering in typologically diverse languages. TACL, 2020. URL https://aclanthology.org/2020.tacl-1.30. ↗ our card
29. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? Try arc, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. URL https: //arxiv.org/abs/1803.05457. find
30. Cobbe, K., Kosaraju, V., Bavarian, M., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. URL https://arxiv.org/abs/2110.14168. ↗ our card
31. Crenshaw, K. Demarginalizing the intersection of race and sex: A black feminist critique of antidiscrimination doctrine, feminist theory and antiracist politics, 1989. find
32. Dai, A. M. and Le, Q. V. Semi-supervised sequence learning. In Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper\_files/paper/2015/file/ 7137deb… ↗
33. Daniels, P. T. and Bright, W. The world’s writing systems. Oxford University Press on Demand, 1996. find
34. Denton, E., Hanna, A., Amironesei, R., Smart, A., Nicole, H., and Scheuerman, M. K. Bringing the people back in: Contesting benchmark machine learning datasets, 2020. find
35. Dev, S., Monajatipoor, M., Ovalle, A., Subramonian, A., Phillips, J. M., and Chang, K.-W. Harms of gender exclusivity and challenges in non-binary representation in language technologies, 2021a. URL https://arxiv.org/abs/ 2108.12084. ↗
36. Dev, S., Sheng, E., Zhao, J., Amstutz, A., Sun, J., Hou, Y., Sanseverino, M., Kim, J., Nishi, A., Peng, N., and Chang, K.-W. On measures of biases and harms in NLP. August 2021b. URL https://arxiv.org/abs/2108.03362. ↗
37. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL, 2019. URL https://aclanthology.org/N19-1423. ↗
38. Diaz, M., Kivlichan, I. D., Rosen, R., Baker, D. K., Amironesei, R., Prabhakaran, V., and Denton, E. CrowdWorkSheets: Accounting for individual and collective identities underlying crowdsourced dataset annotation. June 2022. URL https://arxiv.org/abs/2206.08931. — [arXiv:2206.08931](https://arxiv.org/abs/2206.08931) ↗
39. Dinan, E., Humeau, S., Chintagunta, B., and Weston, J. Build it break it fix it for dialogue safety: Robustness from adversarial human attack. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proc… find our card
40. Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., and Gardner, M. Documenting large webtext corpora: A case study on the colossal clean crawled corpus, 2021. find
41. Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., Zoph, B., Fedus, L., Bosma, M., Zhou, Z., Wang, T., Wang, Y. E., Webster, K., Pellat, M., Robinson, K., Meier-Hellstern, K., Duke, T., Dixon, L., Zhang, K., Le, Q. V., Wu, Y., Chen, Z., and Cui… find our card
42. Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., and Gardner, M. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo… find our card
43. Feldman, V. Does learning require memorization? a short tale about a long tail. corr abs/1906.05271 (2019). arXiv preprint arXiv:1906.05271, 2019. URL https://arxiv.org/abs/1906.05271. — [arXiv:1906.05271](https://arxiv.org/abs/1906.05271) ↗
44. Freitag, M., Foster, G., Grangier, D., Ratnakar, V., Tan, Q., and Macherey, W. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9: 1460–1474, 2021. doi: 10.1162/tacl\_a\_00437. URL https://acla… find our card
45. Freitag, M., Rei, R., Mathur, N., Lo, C.-k., Stewart, C., Avramidis, E., Kocmi, T., Foster, G., Lavie, A., and Martins, A. F. T. Results of WMT22 metrics shared task: Stop using BLEU – neural metrics are better and more robust. In Proceedings of the Seventh Conference on Machine Translation (WMT), p… find our card
46. Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., Jones, A., Bowman, S., Chen, A., Conerly, T., DasSarma, N., Drain, D., Elhage, N., El-Showk, S., Fort, S., Hatfield-Dodds, Z., Henighan, T., Hernandez, D., Hume, T., Jacobson, J.,… find our card
47. Garg, N., Schiebinger, L., Jurafsky, D., and Zou, J. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115(16):E3635–E3644, 2018. doi: 10.1073/pnas.1720347115. URL https://www.pnas.org/doi/abs/10.1073/pnas.1720347115. — [doi:10.1073/pnas.1720347115](https://doi.org/10.1073/pnas.1720347115) ↗
48. Garg, T., Masud, S., Suresh, T., and Chakraborty, T. Handling bias in toxic speech detection: A survey. January 2022. URL https://arxiv.org/abs/2202.00126. — [arXiv:2202.00126](https://arxiv.org/abs/2202.00126) ↗
49. Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., au2, H. D. I., and Crawford, K. Datasheets for datasets, 2021. find
50. Gehman, S., Gururangan, S., Sap, M., Choi, Y., and Smith, N. A. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 3356–3369, Online, November 2020. Association for Computational Linguistics. doi… find our card
51. Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., and Berant, J. Did aristotle use a laptop? A question answering benchmark with implicit reasoning strategies. TACL, 2021. doi: 10.1162/tacl\_a\_00370. URL https://aclanthology.org/2021.tacl-1.21. ↗
52. Github. Your AI pair programmer, October 2021. find
53. Glaese, A., McAleese, N., Tr˛ebacz, M., Aslanides, J., Firoiu, V., Ewalds, T., Rauh, M., Weidinger, L., Chadwick, M., Thacker, P., Campbell-Gillingham, L., Uesato, J., Huang, P.-S., Comanescu, R., Yang, F., See, A., Dathathri, S., Greig, R., Chen, C., Fritz, D., Elias, J. S., Green, R., Mokrá, S., F… find our card
54. Goldfarb-Tarrant, S., Marchant, R., Muñoz Sánchez, R., Pandya, M., and Lopez, A. Intrinsic bias metrics do not correlate with application bias. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language P… find
55. Google. Our principles, 2018. URL https://ai.google/responsibility/principles/. Accessed May 16, 2023. ↗
56. Google. Generative ai prohibited use policy, 2023a. generative-ai/use-policy. Accessed May 16, 2023. find
57. URL https://policies.google.com/terms/Google. Palm api and makersuite additional terms of service, 2023b. URL https://developers.generativeai. google/terms. Accessed May 16, 2023. ↗
58. Goyal, N., Kivlichan, I., Rosen, R., and Vasserman, L. Is your toxicity my toxicity? Exploring the impact of rater identity on toxicity annotation. May 2022. URL https://arxiv.org/abs/2205.00501. — [arXiv:2205.00501](https://arxiv.org/abs/2205.00501) ↗
59. Graves, A. Generating sequences with recurrent neural networks, 2014. find
60. Hanna, A., Denton, E., Smart, A., and Smith-Loud, J. Towards a critical race methodology in algorithmic fairness. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, FAT* ’20, pp. 501–512, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 97814503693… — [doi:10.1145/3351095.3372826](https://doi.org/10.1145/3351095.3372826) find
61. Hasan, T., Bhattacharjee, A., Islam, M. S., Mubasshir, K., Li, Y.-F., Kang, Y.-B., Rahman, M. S., and Shahriyar, R. XL-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 4693–4703, Online, Augus… find
62. Hendricks, L. A., Burns, K., Saenko, K., Darrell, T., and Rohrbach, A. Women also snowboard: Overcoming bias in captioning models (extended abstract), 2018. find
63. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. URL https: //arxiv.org/abs/2103.03874. find our card
64. Hochreiter, S. and Schmidhuber, J. Long Short-Term Memory. Neural Computation, 9(8):1735–1780, 11 1997. ISSN 0899-7667. doi: 10.1162/neco.1997.9.8.1735. URL https://doi.org/10.1162/neco.1997.9.8.1735. ↗
65. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., et al. Training compute-optimal large language models. NeurIPS, 2022. URL https://arxiv.org/abs/2203.15556. ↗
66. Howard, J. and Ruder, S. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 328–339, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.… find
67. Hsiao, S. and Collins, E. Try bard and share your feedback. https://blog.google/technology/ai/try-bard/, March 2023. Accessed: 2023-5-5. ↗
68. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-Rank adaptation of large language models. June 2021. URL https://arxiv.org/abs/2106.09685. ↗
69. Ippolito, D., Tramèr, F., Nasr, M., Zhang, C., Jagielski, M., Lee, K., Choquette-Choo, C. A., and Carlini, N. Preventing verbatim memorization in language models gives a false sense of privacy. arXiv preprint arXiv:2210.17546, 2022. URL https://arxiv.org/abs/210.17546. — [arXiv:2210.17546](https://arxiv.org/abs/2210.17546) ↗
70. Jacobs, A. Z. and Wallach, H. Measurement and fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, pp. 375–385, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383097. doi: 10.1145/3442188.3445901. URL https://doi.org… ↗
71. Jagielski, M., Thakkar, O., Tramer, F., Ippolito, D., Lee, K., Carlini, N., Wallace, E., Song, S., Thakurta, A., Papernot, N., et al. Measuring forgetting of memorized training examples. arXiv preprint arXiv:2207.00099, 2022. URL https://arxiv.org/abs/2207.00099. — [arXiv:2207.00099](https://arxiv.org/abs/2207.00099) ↗
72. Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., and Fung, P. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38, mar 2023. doi: 10.1145/3571730. URL https://doi.org/10.1145%2F3571730. find our card
73. Jigsaw. Toxic comment classification challenge, 2018. jigsaw-toxic-comment-classification-challenge. find
74. URL https://www.kaggle.com/c/Jigsaw. Exploring the role of human raters in creating nlp datasets, 2019a. URL https://medium.com/jigsaw/ creating-labeled-datasets-and-exploring-the-role-of-human-raters-56367b6db298. ↗
75. Jigsaw. Jigsaw multilingual toxic comment classification, 2019b. jigsaw-multilingual-toxic-comment-classification. find
76. URL https://www.kaggle.com/c/Joshi, M., Choi, E., Weld, D., and Zettlemoyer, L. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601–1611,… ↗ our card
77. Jouppi, N. P., Yoon, D. H., Kurian, G., Li, S., Patil, N., Laudon, J., Young, C., and Patterson, D. A domain-specific supercomputer for training deep neural networks. Communications of the ACM, 63(7):67–78, 2020. find
78. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. URL https: //arxiv.org/abs/2001.08361. find
79. Keyes, O. The misgendering machines: Trans/hci implications of automatic gender recognition. Proc. ACM Hum.-Comput. Interact., 2(CSCW), nov 2018. doi: 10.1145/3274357. URL https://doi.org/10.1145/3274357. — [doi:10.1145/3274357](https://doi.org/10.1145/3274357) ↗
80. Kneser, R. and Ney, H. Improved backing-off for m-gram language modeling. In 1995 International Conference on Acoustics, Speech, and Signal Processing, volume 1, pp. 181–184 vol.1, 1995. doi: 10.1109/ICASSP.1995.479394. — [doi:10.1109/icassp.1995.479394](https://doi.org/10.1109/icassp.1995.479394) find
81. Korbak, T., Shi, K., Chen, A., Bhalerao, R., Buckley, C. L., Phang, J., Bowman, S. R., and Perez, E. Pretraining language models with human preferences, 2023. URL https://arxiv.org/abs/2302.08582. ↗
82. Kreutzer, J., Caswell, I., Wang, L., Wahab, A., van Esch, D., Ulzii-Orshikh, N., Tapo, A., Subramani, N., Sokolov, A., Sikasote, C., et al. Quality at a glance: An audit of web-crawled multilingual datasets. Transactions of the Association for Computational Linguistics, 10:50–72, 2022. find our card
83. Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.-W., Dai, A. M., Uszkoreit, J., Le, Q., and Petrov, S. Natural questions: A benchmark for question answering research. T… find our card
84. Ladhak, F., Durmus, E., Cardie, C., and McKeown, K. WikiLingua: A new benchmark dataset for cross-lingual abstractive summarization. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4034– 4048, Online, November 2020. Association for Computational Linguistics. doi: 10.186… find our card
85. Lai, G., Xie, Q., Liu, H., Yang, Y., and Hovy, E. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 785–794, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.… find our card
86. Lee, C. Welcome, singular "they". https://apastyle.apa.org/blog/singular-they, 2019. Accessed: 2022-11-18. ↗
87. Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., and Carlini, N. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021. URL https://arxiv.org/abs/2107. 06499. ↗
88. Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045–3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Lingui… find
89. Levesque, H., Davis, E., and Morgenstern, L. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012. find
90. Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., et al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206.14858, 2022. URL https://arxiv.org/abs/2206.14858. ↗ our card
91. Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., Newman, B., Yuan, B., Yan, B., Zhang, C., Cosgrove, C., Manning, C. D., Ré, C., Acosta-Navas, D., Hudson, D. A., Zelikman, E., Durmus, E., Ladhak, F., Rong, F., Ren, H., Yao, H., Wan… find
92. Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q. V., Zoph, B., Wei, J., and Roberts, A. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301.13688, 2023. find our card
93. Luccioni, A. S. and Viviano, J. D. What’s in the box? a preliminary analysis of undesirable content in the common crawl corpus, 2021. find
94. Meurer, A., Smith, C. P., Paprocki, M., Čertík, O., Kirpichev, S. B., Rocklin, M., Kumar, A., Ivanov, S., Moore, J. K., Singh, S., et al. Sympy: symbolic computing in python. PeerJ Computer Science, 3:e103, 2017. find
95. Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381–2391, Brussels, Belgium, October-November 2018. Associatio… find
96. Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., and Gebru, T. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, pp. 220–229, 2019. find
97. Mostafazadeh, N., Chambers, N., He, X., Parikh, D., Batra, D., Vanderwende, L., Kohli, P., and Allen, J. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics… find
98. Movva, R., Shanmugam, D., Hou, K., Pathak, P., Guttag, J., Garg, N., and Pierson, E. Coarse race data conceals disparities in clinical risk score performance, 2023. find
99. Mozes, M., Hoffmann, J., Tomanek, K., Kouate, M., Thain, N., Yuan, A., Bolukbasi, T., and Dixon, L. Towards agile text classifiers for everyone, 2023. URL https://arxiv.org/abs/2302.06541. ↗
100. Narayan, S., Cohen, S. B., and Lapata, M. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797–1807, Brussels, Belgium, October-November 20… find
101. Nie, Y., Williams, A., Dinan, E., Bansal, M., Weston, J., and Kiela, D. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4885–4901, Online, July 2020. Association for Computational Lin… find
102. OpenAI. Introducing ChatGPT. https://openai.com/blog/chatgpt, November 2022. Accessed: 2023-5-5. ↗
103. OpenAI. ChatGPT plugins. https://openai.com/blog/chatgpt-plugins, March 2023a. Accessed: 2023-5-5. ↗
104. OpenAI. Gpt-4 technical report, March 2023b. URL https://arxiv.org/abs/2303.08774. ↗ our card
105. Orlanski, G., Xiao, K., Garcia, X., Hui, J., Howland, J., Malmaud, J., Austin, J., Singh, R., and Catasta, M. Measuring the impact of programming language distribution. February 2023. find
106. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Gray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with h… find
107. Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N. Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume… find our card
108. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311–318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Lingui… find our card
109. Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., Htut, P. M., and Bowman, S. R. BBQ: A hand-built bias benchmark for question answering. CoRR, abs/2110.08193, 2021. URL https://arxiv.org/abs/ 2110.08193. ↗ our card
110. Paullada, A., Raji, I. D., Bender, E. M., Denton, E., and Hanna, A. Data and its (dis)contents: A survey of dataset development and use in machine learning research. Patterns, 2(11):100336, nov 2021. doi: 10.1016/j.patter.2021. 100336. URL https://doi.org/10.1016%2Fj.patter.2021.100336. find
111. Pax. Pax, 2022. URL https://github.com/google/paxml. ↗
112. Ponti, E. M., Glavaš, G., Majewska, O., Liu, Q., Vulić, I., and Korhonen, A. XCOPA: A multilingual dataset for causal commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2362–2376, Online, November 2020. Association for Compu… find our card
113. Pozzobon, L., Ermis, B., Lewis, P., and Hooker, S. On the challenges of using black-box apis for toxicity evaluation in research, 2023. find
114. Prabhakaran, V., Qadri, R., and Hutchinson, B. Cultural incongruencies in artificial intelligence. November 2022. URL https://arxiv.org/abs/2211.13069. — [arXiv:2211.13069](https://arxiv.org/abs/2211.13069) ↗
115. Prabhu, V. U. and Birhane, A. Large image datasets: A pyrrhic win for computer vision?, 2020. find
116. Pushkarna, M., Zaldivar, A., and Kjartansson, O. Data cards: Purposeful and transparent dataset documentation for responsible ai, 2022. find
117. Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., Rutherford, E., Hennigan, T., Menick, J., Cassirer, A., Powell, R., van den Driessche, G., Hendricks, L. A., Rauh, M., Huang, P.-S., Glaese, A., Welbl, J., Dathathri, S., Huang… find
118. Raji, I. D., Bender, E. M., Paullada, A., Denton, E., and Hanna, A. AI and the everything in the whole wide world benchmark. November 2021. URL https://arxiv.org/abs/2111.15366. ↗
119. Rajpurkar, P., Jia, R., and Liang, P. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784–789, Melbourne, Australia, July 2018. Association for Computational Linguisti… find our card
120. Ramesh, D., Kameswaran, V., Ding, W., and Sambasivan, N. How platform-user power relations shape algorithmic accountability: A case study of instant loan platforms and financially stressed users in india. In FAccT ’22: 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 1917–1928,… find
121. Rauh, M., Mellor, J., Uesato, J., Huang, P.-S., Welbl, J., Weidinger, L., Dathathri, S., Glaese, A., Irving, G., Gabriel, I., Isaac, W., and Hendricks, L. A. Characteristics of harmful text: Towards rigorous benchmarking of language models, 2022. find
122. Replit. Meet replit ghostwriter, your partner in code. https://blog.replit.com/ghostwriter, October 2022. Accessed: 2023-5-5. ↗
123. Riley, P., Dozat, T., Botha, J. A., Garcia, X., Garrette, D., Riesa, J., Firat, O., and Constant, N. Frmt: A benchmark for few-shot region-aware machine translation. Transactions of the Association for Computational Linguistics, 2023. find
124. Roberts, A., Raffel, C., and Shazeer, N. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5418– 5426, Online, November 2020. Association for Computational Linguistics. doi:… find
125. Rodriguez, P., Barrow, J., Hoyle, A. M., Lalor, J. P., Jia, R., and Boyd-Graber, J. Evaluation examples are not equally informative: How should that change NLP leaderboards? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Co… find
126. Ruder, S., Vulić, I., and Søgaard, A. Square one bias in NLP: Towards a multi-dimensional exploration of the research manifold. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 2340–2354, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/… find
127. Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021. find
128. Sambasivan, N., Arnesen, E., Hutchinson, B., Doshi, T., and Prabhakaran, V. Re-imagining algorithmic fairness in india and beyond. In FAccT ’21: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 315–328, March 2021. find
129. Sap, M., Gabriel, S., Qin, L., Jurafsky, D., Smith, N. A., and Choi, Y. Social bias frames: Reasoning about social and power implications of language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5477–5490, Online, July 2020. Association for Computa… find our card
130. Sap, M., Swayamdipta, S., Vianna, L., Zhou, X., Choi, Y., and Smith, N. A. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. November 2021. URL https://arxiv.org/abs/2111.07997. ↗ our card
131. Sax. Sax, 2022. URL https://github.com/google/saxml. ↗
132. Schick, T., Udupa, S., and Schütze, H. Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in NLP. CoRR, abs/2103.00453, 2021. URL https://arxiv.org/abs/2103.00453. ↗ our card
133. Schlangen, D. Targeting the benchmark: On methodology in current natural language processing research, 2020. find
134. Selbst, A. D., Boyd, D., and Friedler, S. A. Fairness and abstraction in sociotechnical systems. In FFAT* ’19: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68, January 2019. find
135. Sellam, T., Das, D., and Parikh, A. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7881–7892, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.704. URL ht… find
136. Shannon, C. E. Prediction and entropy of printed english. Bell System Technical Journal, 30(1):50–64, 1951. doi: https://doi.org/10.1002/j.1538-7305.1951.tb01366.x. URL https://onlinelibrary.wiley.com/doi/abs/10. 1002/j.1538-7305.1951.tb01366.x. — [doi:10.1002/j.1538-7305.1951.tb01366.x](https://doi.org/10.1002/j.1538-7305.1951.tb01366.x) ↗
137. Shelby, R., Rismani, S., Henne, K., Moon, A., Rostamzadeh, N., Nicholas, P., Yilla, N., Gallegos, J., Smart, A., Garcia, E., and Virk, G. Identifying sociotechnical harms of algorithmic systems: Scoping a taxonomy for harm reduction, 2023. URL https://arxiv.org/abs/2210.05791. — [arXiv:2210.05791](https://arxiv.org/abs/2210.05791) ↗
138. Shi, F., Suzgun, M., Freitag, M., Wang, X., Srivats, S., Vosoughi, S., Chung, H. W., Tay, Y., Ruder, S., Zhou, D., Das, D., and Wei, J. Language Models are Multilingual Chain-of-Thought Reasoners. In Proceedings of ICLR 2023, 2023. URL http://arxiv.org/abs/2210.03057. ↗ our card
139. Smith, E. M., Hall, M., Kambadur, M., Presani, E., and Williams, A. “i’m sorry to hear that”: Finding new biases in language models with a holistic descriptor dataset. May 2022. URL https://arxiv.org/abs/2205.09209. ↗
140. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022. URL https://arxiv.org/abs/… ↗ our card
141. Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., et al. Challenging BIG-Bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022. URL https://arxiv.org/abs/2210.09261. ↗ our card
142. Tabachnyk, M. and Nikolov, S. {ML-Enhanced} code completion improves developer productivity. July 2022. find
143. Talmor, A., Herzig, J., Lourie, N., and Berant, J. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Shor… find
144. Tay, Y., Dehghani, M., Tran, V. Q., Garcia, X., Wei, J., Wang, X., Chung, H. W., Bahri, D., Schuster, T., Zheng, S., Zhou, D., Houlsby, N., and Metzler, D. UL2: Unifying language learning paradigms. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.ne… ↗ our card
145. Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. LaMDA: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022. URL https://arxiv.org/abs/2201.08239. ↗ our card
146. Tomasev, N., McKee, K. R., Kay, J., and Mohamed, S. Fairness for unobserved characteristics: Insights from technological impacts on queer communities. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. ACM, jul 2021. doi: 10.1145/3461702.3462540. URL https://doi.org/10.1145%2… find
147. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, v… find
148. Vilar, D., Freitag, M., Cherry, C., Luo, J., Ratnakar, V., and Foster, G. Prompting palm for translation: Assessing strategies and performance. arXiv preprint arXiv:2211.09102, 2022. URL https://arxiv.org/abs/2211.09102. ↗
149. Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. SuperGlue: A stickier benchmark for general-purpose language understanding systems. NeurIPS, 2019. URL http://go/arxiv/ 1905.00537. find our card
150. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In Proceedings of ICLR 2023, 2023. find our card
151. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS, 2022. URL https://arxiv.org/abs/2201. 11903. ↗ our card
152. Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., Kenton, Z., Brown, S., Hawkins, W., Stepleton, T., Biles, C., Birhane, A., Haas, J., Rimell, L., Hendricks, L. A., Isaac, W., Legassick, S., Irving, G., and Gabriel, I. Ethic… find our card
153. Welty, C., Paritosh, P., and Aroyo, L. Metrology for ai: From benchmarks to instruments, 2019. find
154. Xu, A., Pathak, E., Wallace, E., Gururangan, S., Sap, M., and Klein, D. Detoxifying language models risks marginalizing minority voices. April 2021a. URL https://arxiv.org/abs/2104.06390. ↗
155. Xu, Y., Lee, H., Chen, D., Hechtman, B., Huang, Y., Joshi, R., Krikun, M., Lepikhin, D., Ly, A., Maggioni, M., et al. Gspmd: general and scalable parallelization for ml computation graphs. arXiv preprint arXiv:2105.04663, 2021b. — [arXiv:2105.04663](https://arxiv.org/abs/2105.04663) find
156. Xu, Y., Zhu, C., Wang, S., Sun, S., Cheng, H., Liu, X., Gao, J., He, P., Zeng, M., and Huang, X. Human parity on CommonsenseQA: Augmenting self-attention with external attention. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (IJCAI-22), 2022. find
157. Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., and Raffel, C. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language… find
158. Yin, P., Li, W.-D., Xiao, K., Rao, A., Wen, Y., Shi, K., Howland, J., Bailey, P., Catasta, M., Michalewski, H., Polozov, A., and Sutton, C. Natural language to code generation in interactive data science notebooks. December 2022. find
159. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791–4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi:… find