Elamur Research⌕
☰
home / papers / 2022 / AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model

AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model

Paper · 2022 · Saleh Soltan, Shankar Ananthakrishnan, Jack FitzGerald et al.

Summary

Large language models that learn from a few examples in the prompt have mostly been decoder-only. Their left-to-right attention can be a handicap when the input is long. This Amazon Alexa AI paper asks whether an encoder-decoder model can be an equally strong in-context learner.

The authors train AlexaTM 20B, a Pre-LN Transformer seq2seq model with about 20 billion parameters. It was trained on roughly one trillion tokens of Wikipedia and mC4 text in 12 languages. Pre-training combines two tasks: span denoising without mask tokens (80% of steps) and a Causal Language Modeling continuation task (20%). The encoder was initialized from an earlier 10B encoder.

Although much smaller than its rivals, the model matches or beats PaLM 540B on one-shot summarization. It also sets new one-shot translation results on Flores-101, especially for low-resource and non-English pairs, and surpasses GPT3 175B zero-shot on SuperGLUE and SQuADv2. The work suggests that mixed-objective seq2seq pre-training is a cheaper and competitive route to multilingual LLMs. Ukrainian is not among the covered languages.

Key points

  • Data and setup
    • Wikipedia supplies 119B tokens (9%) and mC4 supplies 1.2T tokens (91%).
    • Languages are rebalanced with α = 0.5 sampling, and Wikipedia is upsampled.
    • Text is mixed 7:3 in favor of a spoken-style format.
    • Tokenization uses a 150K unigram SentencePiece vocabulary.
  • Denoising objective: 15% of tokens are dropped in Poisson-length spans (mean 3), with no [MASK] symbols. A [CLM] token tells the model to continue a text instead of reconstructing it.
  • Summarization and translation
    • One-shot results are on par with or better than PaLM 540B on XSUM and MLSum (de, es).
    • On Flores-101, Arabic→Tamil reaches 21.8 spBLEU, compared with 0.9 for supervised M2M-124 615M.
  • Zero-shot multilingual tasks: the model achieves the best reported results on XNLI, XCOPA, PAWS-X and XWinograd, beating XGLM 7.5B. It also leads on the Winogender bias benchmark.
  • Risks and cost
    • Memorization drops slightly as context length grows.
    • Toxic prompts still raise the toxicity of continuations.
    • Training used 15,360 A100 GPU-days and about 98.2 tCO2e.
    • Weights are promised at github.com/amazon-research/alexa-teacher-models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksMultilingual & cross-lingual NLP · 882 works

Cite

Cite as Saleh Soltan, Shankar Ananthakrishnan, Jack FitzGerald, Rahul Gupta, Wael Hamza, Haidar Khan, Charith Peris, Stephen Rawls, Andy Rosenbaum, Anna Rumshisky, Chandana Satya Prakash, Mukund Sridhar, Fabian Triefenbach, Apurv Verma, Gokhan Tur and Prem Natarajan. 2022. AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model. arXiv:2208.01448.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2208.01448 · arXiv 2208.01448

@misc{soltan2022alexatm,
  title = {{AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model}},
  author = {Saleh Soltan, Shankar Ananthakrishnan, Jack FitzGerald, Rahul Gupta, Wael Hamza, Haidar Khan, Charith Peris, Stephen Rawls, Andy Rosenbaum, Anna Rumshisky, Chandana Satya Prakash, Mukund Sridhar, Fabian Triefenbach, Apurv Verma, Gokhan Tur and Prem Natarajan},
  year = {2022},
  doi = {10.48550/arxiv.2208.01448},
  eprint = {2208.01448},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2208.01448}
}

References

88 references, 55 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bakshi, S., Batra, S., Heidari, P., Arun, A., Jain, S., and White, M. Structure-to-text generation with self-training, acceptability classifiers and context-conditioning for the gem shared task. In GEM, 2021. find
  2. Bao, H., Dong, L., Wei, F., Wang, W., Yang, N., Liu, X., Wang, Y., Piao, S., Gao, J., Zhou, M., and Hon, H.-W. Unilmv2: Pseudo-masked language models for unified language model pre-training. ArXiv, abs/2002.12804, 2020.
  3. Blodgett, S. L., Barocas, S., Daumé III, H., and Wallach, H. Language (technology) is power: A critical survey of “bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5454–5476, Online, July 2020. Association for Computational Linguistics. d…
  4. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess… find
  5. Curran Associates, Inc., 2020a. URL https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf. ↗
  6. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T. J., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S.,… — Language Models are Few-Shot Learners
  7. Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramèr, F., and Zhang, C. Quantifying memorization across neural language models. ArXiv, abs/2202.07646, 2022. find
  8. Chiu, J. and Zheng, S. Making deepspeed zero run efficiently on more-affordable hardware, Mar 2022. URL https://www.amazon.science/blog/making-deepspeed-zero-run-efficiently-on-more-affordable-hardware. ↗
  9. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A. B., Barnes, P., Tay, Y., Shazeer, N. M., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B. C., Pope, R., Bradbury,… — PaLM: Scaling Language Modeling with Pathways
  10. Conneau, A., Lample, G., Rinott, R., Williams, A., Bowman, S. R., Schwenk, H., and Stoyanov, V. Xnli: Evaluating cross-lingual sentence representations. In EMNLP, 2018. find
  11. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. Unsupervised cross-lingual representation learning at scale. In ACL, 2020.
  12. Dathathri, S., Madotto, A., Lan, J., Hung, J., Frank, E., Molino, P., Yosinski, J., and Liu, R. Plug and play language models: A simple approach to controlled text generation, 2019.
  13. Davy, B. Building an aws ec2 carbon emissions datase. https://medium.com/teads-engineering/building-an-aws-ec2-carbon-emissions-dataset-3f0fd76c98ac, 2021. ↗
  14. Dev, S., Li, T., Phillips, J. M., and Srikumar, V. On Measuring and Mitigating Biased Inferences of Word Embeddings. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7659–7666, April 2020. doi: 10.1609/aaai.v34i05.6267. URL https://ojs.aaai.org/index.php/AAAI/article/view/6267. — [doi:10.1609/aaai.v34i05.6267](https://doi.org/10.1609/aaai.v34i05.6267) ↗
  15. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and…
  16. Dhamala, J., Sun, T., Kumar, V., Krishna, S., Pruksachatkun, Y., Chang, K.-W., and Gupta, R. Bold: Dataset and metrics for measuring biases in open-ended language generation. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 2021. find
  17. Dinan, E., Humeau, S., Chintagunta, B., and Weston, J. Build it break it fix it for dialogue safety: Robustness from adversarial human attack. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proc…
  18. Dinan, E., Fan, A., Williams, A., Urbanek, J., Kiela, D., and Weston, J. Queens are powerful too: Mitigating gender bias in dialogue generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8173–8188, Online, November 2020. Association for Co…
  19. Dinan, E., Abercrombie, G., Bergman, A. S., Spruit, S. L., Hovy, D., Boureau, Y.-L., and Rieser, V. Anticipating safety issues in e2e conversational ai: Framework and tooling. ArXiv, abs/2107.03451, 2021. — [arXiv:2107.03451](https://arxiv.org/abs/2107.03451) find
  20. Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., and Hon, H.-W. Unified language model pre-training for natural language understanding and generation. ArXiv, abs/1905.03197, 2019.
  21. Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., Zoph, B., Fedus, L., Bosma, M., Zhou, Z., Wang, T., Wang, Y. E., Webster, K., Pellat, M., Robinson, K., Meier-Hellstern, K. S., Duke, T., Dixon, L., Zhang, K., Le, Q. V., Wu, Y., Chen, Z., and… — GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
  22. Dusek, O., Howcroft, D. M., and Rieser, V. Semantic noise matters for neural natural language generation. ArXiv, abs/1911.03905, 2019.
  23. Fan, A., Bhosale, S., Schwenk, H., Ma, Z., El-Kishky, A., Goyal, S., Baines, M., Çelebi, O., Wenzek, G., Chaudhary, V., Goyal, N., Birch, T., Liptchinsky, V., Edunov, S., Grave, E., Auli, M., and Joulin, A. Beyond english-centric multilingual machine translation. J. Mach. Learn. Res., 22:107:1–107:… find
  24. Ferreira, T. C., Gardent, C., Ilinykh, N., van der Lee, C., Mille, S., Moussallem, D., and Shimorina, A. The 2020 bilingual, bi-directional webnlg+ shared task: Overview and evaluation results (webnlg+ 2020). In WEBNLG, 2020.
  25. FitzGerald, J. G. M., Ananthakrishnan, S., Arkoudas, K., Bernardi, D., Bhagia, A., Bovi, C. D., Cao, J., Chada, R., Chauhan, A., Chen, L., Dwarakanath, A., Dwivedi, S., Gojayev, T., Gopalakrishnan, K., Gueudré, T., Hakkani-Tür, D. Z., Hamza, W., Hueser, J., Jose, K. M., Khan, H., Liu, B., Lu, J.,… — [arXiv:2206.07808](https://arxiv.org/abs/2206.07808) find
  26. Gehman, S., Gururangan, S., Sap, M., Choi, Y., and Smith, N. A. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. Findings of the Association for Computational Linguistics: EMNLP 2020, 2020. doi: 10.18653/v1/2020.findings-emnlp.301. URL http://dx.doi.org/10.18653/v1/2020.… ↗
  27. Gehrmann, S., Adewumi, T. P., Aggarwal, K., Ammanamanchi, P. S., Anuoluwapo, A., Bosselut, A., Chandu, K. R., Clinciu, M., Das, D., Dhole, K. D., Du, W., Durmus, E., Dusek, O., Emezue, C. C., Gangal, V., Garbacea, C., Hashimoto, T. B., Hou, Y., Jernite, Y., Jhamtani, H., Ji, Y., Jolly, S., Kale, M.,… — The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
  28. Goyal, N., Gao, C., Chaudhary, V., Chen, P.-J., Wenzek, G., Ju, D., Krishnan, S., Ranzato, M., Guzmán, F., and Fan, A. The flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10:522–538, 2022. find
  29. Gupta, U., Dhamala, J., Kumar, V., Verma, A., Pruksachatkun, Y., Krishna, S., Gupta, R., Chang, K.-W., Ver Steeg, G., and Galstyan, A. Mitigating gender bias in distilled language models via counterfactual role reversal. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 658… find
  30. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O., and Sifre, L.… — Training Compute-Optimal Large Language Models
  31. Honnibal, M. and Montani, I. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear, 2017. find
  32. Izacard, G. and Grave, E. Leveraging passage retrieval with generative models for open domain question answering. In EACL, 2021.
  33. Jagielski, M., Ullman, J., and Oprea, A. Auditing differentially private machine learning: How private is private sgd? ArXiv, abs/2006.07709, 2020. — [arXiv:2006.07709](https://arxiv.org/abs/2006.07709) find
  34. Jayaraman, B. and Evans, D. Evaluating differentially private machine learning in practice. In 28th USENIX Security Symposium (USENIX Security 19), pp. 1895–1912, Santa Clara, CA, August 2019. USENIX Association. ISBN 978-1-939133-06-9. URL https://www.usenix.org/conference/usenixsecurity19/presenta… ↗
  35. Kandpal, N., Wallace, E., and Raffel, C. Deduplicating training data mitigates privacy risks in language models. ArXiv, abs/2202.06539, 2022.
  36. Khrushchev, M. Yandex publishes yalm 100b. it’s the largest gpt-like neural network in open source. https://medium.com/yandex/yandex-publishes-yalm-100b-its-the-largest-gpt-like-neural-network-in-open-source-d1df53d0e9a6, 2022. ↗
  37. Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2015.
  38. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners, 2022. URL https://arxiv.org/abs/2205.11916. ↗
  39. Kudo, T. and Richardson, J. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In EMNLP, 2018.
  40. Kurita, K., Vyas, N., Pareek, A., Black, A. W., and Tsvetkov, Y. Quantifying social biases in contextual word representations. 1st ACL Workshop on Gender Bias for Natural Language Processing. URL https://par.nsf.gov/biblio/10098355. Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Bu… ↗
  41. Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. ArXiv, abs/2104.08691, 2021.
  42. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In ACL, 2020.
  43. Lhoest, Q., Villanova del Moral, A., Jernite, Y., Thakur, A., von Platen, P., Patil, S., Chaumond, J., Drame, M., Plu, J., Tunstall, L., Davison, J., Šaško, M., Chhablani, G., Malik, B., Brandeis, S., Le Scao, T., Sanh, V., Xu, C., Patry, N., McMillan-Major, A., Schmid, P., Gugger, S., Delangue, C… — Datasets: A Community Library for Natural Language Processing
  44. Li, X. L. and Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), abs/2101.00190, 2021. find
  45. Liang, P. P., Wu, C., Morency, L.-P., and Salakhutdinov, R. Towards understanding and mitigating social biases in language models. In ICML, 2021.
  46. Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In ACL 2004, 2004.
  47. Lin, X. V., Mihaylov, T., Artetxe, M., Wang, T., Chen, S., Simig, D., Ott, M., Goyal, N., Bhosale, S., Du, J., Pasunuru, R., Shleifer, S., Koura, P. S., Chaudhary, V., O’Horo, B., Wang, J., Zettlemoyer, L., Kozareva, Z., Diab, M., Stoyanov, V., and Li, X. Few-shot learning with multilingual language… — Few-shot Learning with Multilingual Language Models
  48. Liu, H., Dacon, J., Fan, W., Liu, H., Liu, Z., and Tang, J. Does gender matter? towards fairness in dialogue systems. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 4403–4416, Barcelona, Spain (Online), December 2020a. International Committee on Computational L…
  49. Liu, Y., Gu, J., Goyal, N., Li, X., Edunov, S., Ghazvininejad, M., Lewis, M., and Zettlemoyer, L. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742, 2020b. find
  50. Majmudar, J., Dupuy, C., Peris, C. S., Smaili, S., Gupta, R., and Zemel, R. S. Differentially private decoding in large language models. ArXiv, abs/2205.13621, 2022. find
  51. Narayan, S., Cohen, S. B., and Lapata, M. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In EMNLP, 2018. — Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization
  52. Nasr, M., Song, S., Thakurta, A., Papernot, N., and Carlini, N. Adversary instantiation: Lower bounds for differentially private machine learning. 2021 IEEE Symposium on Security and Privacy (SP), pp. 866–882, 2021. find
  53. Novikova, J., Dusek, O., and Rieser, V. The e2e dataset: New challenges for end-to-end generation. In SIGDIAL Conference, 2017.
  54. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L. E., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. J. Training language models to follow instruct… — Training language models to follow instructions with human feedback
  55. Patterson, D. A., Gonzalez, J., Le, Q. V., Liang, C., Munguia, L., Rothchild, D., So, D. R., Texier, M., and Dean, J. Carbon emissions and large neural network training. CoRR, abs/2104.10350, 2021. URL https://arxiv.org/abs/2104.10350. ↗
  56. Ponti, E., Glavavs, G., Majewska, O., Liu, Q., Vulic, I., and Korhonen, A. Xcopa: A multilingual dataset for causal commonsense reasoning. In EMNLP, 2020.
  57. Radford, A. and Narasimhan, K. Improving language understanding by generative pre-training. 2018. find
  58. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. 2019. find
  59. Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., Rutherford, E., Hennigan, T., Menick, J., Cassirer, A., Powell, R., van den Driessche, G., Hendricks, L. A., Rauh, M., Huang, P.-S., Glaese, A., Welbl, J., Dathathri, S., Huang… find
  60. Raffel, C., Shazeer, N. M., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. ArXiv, abs/1910.10683, 2020.
  61. Rajpurkar, P., Jia, R., and Liang, P. Know what you don’t know: Unanswerable questions for squad. In ACL, 2018. — Know What You Don't Know: Unanswerable Questions for SQuAD
  62. Rasley, J., Rajbhandari, S., Ruwase, O., and He, Y. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’20, pp. 3505–3506, New York, NY, USA, 2… — [doi:10.1145/3394486.3406703](https://doi.org/10.1145/3394486.3406703) find
  63. Roller, S., Dinan, E., Goyal, N., Ju, D., Williamson, M., Liu, Y., Xu, J., Ott, M., Smith, E. M., Boureau, Y.-L., and Weston, J. Recipes for building an open-domain chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, p…
  64. Roy, S. and Roth, D. Solving general arithmetic word problems, 2016. URL https://arxiv.org/abs/1608.01413. ↗
  65. Rudinger, R., Naradowsky, J., Leonard, B., and Van Durme, B. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 8–14, New Orleans, Louis…
  66. Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L. A., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., Dey, M., BARI, M. S., Xu, C., Thakker, U., Sharma, S., Szczechla, E., Kim, T., Chhablani, G., Nayak, N. V., Datta, D., Chang, J., Jiang, M. T.-J., Wang, H., Manica, M., She… — Multitask Prompted Training Enables Zero-Shot Task Generalization
  67. Schick, T., Udupa, S., and Schütze, H. Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in NLP. Transactions of the Association for Computational Linguistics, 9:1408–1424, 2021. doi: 10.1162/tacl a 00434. URL https://aclanthology.org/2021.tacl-1.84. ↗
  68. Scialom, T., Dray, P.-A., Lamprier, S., Piwowarski, B., and Staiano, J. Mlsum: The multilingual summarization corpus. ArXiv, abs/2004.14900, 2020.
  69. Sheng, E., Chang, K.-W., Natarajan, P., and Peng, N. The woman worked as a babysitter: On biases in language generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),…
  70. Sheng, E., Chang, K.-W., Natarajan, P., and Peng, N. Societal biases in language generation: Progress and challenges. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long…
  71. Shoeybi, M., Patwary, M. A., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-lm: Training multi-billion parameter language models using model parallelism. ArXiv, abs/1909.08053, 2019. find
  72. Song, K., Tan, X., Qin, T., Lu, J., and Liu, T.-Y. Mass: Masked sequence to sequence pre-training for language generation. In ICML 2019, June 2019. URL https://www.microsoft.com/en-us/research/publication/mass-masked-sequence-to-sequence-pre-training-for-language-generation/. ↗
  73. Tay, Y., Dehghani, M., Tran, V. Q., Garcı́a, X., Bahri, D., Schuster, T., Zheng, H., Houlsby, N., and Metzler, D. Unifying language learning paradigms. ArXiv, abs/2205.05131, 2022.
  74. Tikhonov, A. and Ryabinin, M. It’s all in the heads: Using attention heads as a baseline for cross-lingual transfer in commonsense reasoning. ArXiv, abs/2106.12066, 2021.
  75. Vaswani, A., Shazeer, N. M., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. ArXiv, abs/1706.03762, 2017.
  76. Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. Superglue: A stickier benchmark for general-purpose language understanding systems. In NeurIPS, 2019.
  77. Wang, T., Roberts, A., Hesslow, D., Scao, T. L., Chung, H. W., Beltagy, I., Launay, J., and Raffel, C. What language model architecture and pretraining objective work best for zero-shot generalization? In ICML, 2022. find
  78. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. Chain of thought prompting elicits reasoning in large language models, 2022. URL https://arxiv.org/abs/2201.11903. ↗
  79. Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., Biles, C., Brown, S., Kenton, Z., Hawkins, W., Stepleton, T., Birhane, A., Hendricks, L. A., Rimell, L., Isaac, W., Haas, J., Legassick, S., Irving, G., and Gabriel, I. Taxon… — [doi:10.1145/3531146.3533088](https://doi.org/10.1145/3531146.3533088) find
  80. Welbl, J., Glaese, A., Uesato, J., Dathathri, S., Mellor, J., Hendricks, L. A., Anderson, K., Kohli, P., Coppin, B., and Huang, P.-S. Challenges in detoxifying language models. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 2447–2469, Punta Cana, Dominican Republic, No…
  81. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., and Brew, J. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771, 2019.
  82. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. Transformers: State-of-the-art natural lan… — Transformers: State-of-the-Art Natural Language Processing
  83. Xia, Y., Tan, X., Tian, F., Gao, F., He, D., Chen, W., Fan, Y., Gong, L., Leng, Y., Luo, R., Wang, Y., Wu, L., Zhu, J., Qin, T., and Liu, T.-Y. Microsoft research asia’s systems for wmt19. In WMT, 2019. find
  84. Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L., and Liu, T.-Y. On layer normalization in the transformer architecture. In ICML, 2020. find
  85. Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., and Raffel, C. mt5: A massively multilingual pre-trained text-to-text transformer. In NAACL, 2021.
  86. Yang, Y., Zhang, Y., Tar, C., and Baldridge, J. Paws-x: A cross-lingual adversarial dataset for paraphrase identification. In EMNLP, 2019.
  87. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., Mihaylov, T., Ott, M., Shleifer, S., Shuster, K., Simig, D., Koura, P. S., Sridhar, A., Wang, T., and Zettlemoyer, L. Opt: Open pre-trained transformer language models. ArXiv, abs/2205.01068,…
  88. Zoph, B., Bello, I., Kumar, S., Du, N., Huang, Y., Dean, J., Shazeer, N. M., and Fedus, W. St-moe: Designing stable and transferable sparse expert models. 2022. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/soltan-2022-alexatm-20b-few-shot-learning/