Elamur Research⌕
☰
home / papers / 2023 / Efficiently Adapting Pretrained Language Models To New Languages

Efficiently Adapting Pretrained Language Models To New Languages

Paper · 2023 · Zoltan Csaki, Pian Pawakapan, Urmish Thakker et al.

Summary

Open LLMs trained mostly on English often do poorly on lower-resource languages, and building a model for such a language from scratch requires more data and compute than is usually available. The authors propose a low-cost way to adapt an existing English LLM, a 13B GPT-2-style model, to Hungarian and Thai. The recipe has two parts. First, about 5,000 of the least useful tokens in the original vocabulary are replaced with tokens from a BPE tokenizer trained on the target language, and their embeddings are re-initialized. This keeps the vocabulary size unchanged while sharply reducing the number of tokens needed per word. Second, both continued pretraining and instruction tuning use bilingual data mixtures that combine English with the new language, which limits catastrophic forgetting. The adapted models keep their English benchmark scores roughly unchanged and improve substantially on the new languages. They match or beat open baselines such as Llama2-7B, XGLM-7.5B, mt0-xxl, PULI-GPT-3SX and openthaigpt-7b-chat. The approach matters for any team that wants a capable model for a lower-resource language without paying for full pretraining.

Key points

  • Tokenizer change: About 10% of the 50k GPT-2 vocabulary is swapped for target-language tokens. Section 2.1 reports fertility gains of 42% for Hungarian and 73% for Thai, while the conclusion gives roughly 50% and 70%. This translates to about 2x faster training and inference for Hungarian and 3x for Thai, with little loss of English fertility.
  • Main results: The HU PT + IT model reaches 54.9% on Hungarian multiple-choice, 64.3 F1 on QA and 9.2 ROUGE-2 on summarization, while scoring 58.0% on English multiple-choice. The TH PT + IT model reaches 49.9% on Thai multiple-choice, 48.9 F1 on QA and 12.5 BLEU on translation.
  • Tokenizer ablation: Compared with the original GPT-2 tokenizer, the bilingual tokenizer equals or improves English and most Hungarian scores, for example 54.9% vs 50.8% on HU multiple-choice. Hungarian translation BLEU, however, drops from 8.8 to 6.1.
  • Pretraining mix: Including English in the training data reduces forgetting and also helps Hungarian performance. Results with 25% and 50% English are about the same. Adapting the existing model for 30k steps beats a Hungarian model trained from scratch for 100k steps.
  • Instruction-tuning mix: With no Hungarian instruction data, the model forgets Hungarian. Adding Hungarian helps, but the gains flatten once Hungarian makes up more than about 1% of the 2B-token instruction set.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Zoltan Csaki, Pian Pawakapan, Urmish Thakker and Qiantong Xu. 2023. Efficiently Adapting Pretrained Language Models To New Languages. arXiv:2311.05741.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2311.05741 · arXiv 2311.05741

@misc{csaki2023efficiently,
  title = {{Efficiently Adapting Pretrained Language Models To New Languages}},
  author = {Zoltan Csaki, Pian Pawakapan, Urmish Thakker and Qiantong Xu},
  year = {2023},
  doi = {10.48550/arxiv.2311.05741},
  eprint = {2311.05741},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.05741}
}

References

78 references, 40 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. B. Workshop, “Bloom: A 176b-parameter open-access multilingual language model,” 2023. find
  2. N. Muennighoff, T. Wang, L. Sutawika, A. Roberts, S. Biderman, T. L. Scao, M. S. Bari, S. Shen, Z.-X. Yong, H. Schoelkopf, X. Tang, D. Radev, A. F. Aji, K. Almubarak, S. Albanie, Z. Alyafeai, A. Webson, E. Raff, and C. Raffel, “Crosslingual generalization through multitask finetuning,” 2023.
  3. L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel, “mt5: A massively multilingual pre-trained text-to-text transformer,” 2021.
  4. A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, “Unsupervised cross-lingual representation learning at scale,” 2020. find
  5. N. Goyal, J. Du, M. Ott, G. Anantharaman, and A. Conneau, “Larger-scale transformers for multilingual masked language modeling,” 2021.
  6. O. Shliazhko, A. Fenogenova, M. Tikhonova, V. Mikhailov, A. Kozlova, and T. Shavrina, “mgpt: Few-shot learners go multilingual,” 2022.
  7. X. V. Lin, T. Mihaylov, M. Artetxe, T. Wang, S. Chen, D. Simig, M. Ott, N. Goyal, S. Bhosale, J. Du, R. Pasunuru, S. Shleifer, P. S. Koura, V. Chaudhary, B. O’Horo, J. Wang, L. Zettlemoyer, Z. Kozareva, M. Diab, V. Stoyanov, and X. Li, “Few-shot learning with multilingual language models,” 2022.
  8. H. Xu, B. V. Durme, and K. Murray, “Bert, mbert, or bibert? a study on contextualized embeddings for neural machine translation,” 2021. find
  9. Z.-X. Yong, H. Schoelkopf, N. Muennighoff, A. F. Aji, D. I. Adelani, K. Almubarak, M. S. Bari, L. Sutawika, J. Kasai, A. Baruwa, G. I. Winata, S. Biderman, E. Raff, D. Radev, and V. Nikoulina, “Bloom+1: Adding language support to bloom for zero-shot prompting,” 2023.
  10. J. Phang, I. Calixto, P. M. Htut, Y. Pruksachatkun, H. Liu, C. Vania, K. Kann, and S. R. Bowman, “English intermediate-task training improves zero-shot cross-lingual transfer too,” 2020.
  11. A. Ebrahimi and K. Kann, “How to adapt your pretrained multilingual model to 1600 languages,” 2021.
  12. J. Ye, X. Tao, and L. Kong, “Language versatilists vs. specialists: An empirical revisiting on multilingual transfer ability,” 2023. find
  13. J. Armengol-Estapé, O. de Gibert Bonet, and M. Melero, “On the multilingual capabilities of very large-scale english language models,” 2021. find
  14. K. Ogueji, Y. Zhu, and J. Lin, “Small data? no problem! exploring the viability of pretrained multilingual language models for low-resourced languages,” in Proceedings of the 1st Workshop on Multilingual Representation Learning, 2021, pp. 116–126.
  15. P. Gage, “A new algorithm for data compression,” The C Users Journal Archive, vol. 12, pp. 23–38, 1994. [Online]. Available: https://api.semanticscholar.org/CorpusID:59804030 ↗
  16. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019. find
  17. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Ber… — Language Models are Few-Shot Learners
  18. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models,” 2023.
  19. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan,… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  20. P. Rust, J. Pfeiffer, I. Vulić, S. Ruder, and I. Gurevych, “How good is your tokenizer? on the monolingual performance of multilingual language models,” 2021.
  21. F. Stollenwerk, “Training and evaluation of a multilingual tokenizer for gpt-sw3,” 2023. find
  22. J. Ács. (2019, February) Exploring bert’s vocabulary. https://juditacs.github.io/2019/02/19/bert-tokenization-stats.html [Online]. Available: ↗
  23. R. M. French, “Catastrophic forgetting in connectionist networks,” Trends in cognitive sciences, vol. 3, no. 4, pp. 128–135, 1999. find
  24. S. Cahyawijaya, H. Lovenia, T. Yu, W. Chung, and P. Fung, “Instruct-align: Teaching novel languages with to llms through alignment-based cross-lingual instruction,” 2023. find
  25. I. Chalkidis, M. Fergadiotis, and I. Androutsopoulos, “Multieurlex – a multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer,” 2021. — MultiEURLEX - A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer
  26. T. Vu, A. Barua, B. Lester, D. Cer, M. Iyyer, and N. Constant, “Overcoming catastrophic forgetting in zero-shot cross-lingual generation,” 2022. find
  27. J. Pfeiffer, I. Vulić, I. Gurevych, and S. Ruder, “MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Online: Association for Computational Linguistics, Nov. "2020", pp. 7654–76…
  28. H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. A. Raffel, “Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning,” Advances in Neural Information Processing Systems, vol. 35, pp. 1950–1965, 2022.
  29. R. Pires, H. Abonizio, T. S. Almeida, and R. Nogueira, “Sabiá: Portuguese large language models,” 2023. find
  30. N. Sengupta, S. K. Sahu, B. Jia, S. Katipomu, H. Li, F. Koto, O. M. Afzal, S. Kamboj, O. Pandit, R. Pal et al., “Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models,” arXiv preprint arXiv:2308.16149, 2023.
  31. D. M. Nemeskey, “Natural language processing methods for language modeling,” Ph.D. dissertation, Eötvös Loránd University, 2020. [Online]. Available: https://hlt.bme.hu/media/pdf/nemeskey_thesis.pdf ↗
  32. J. Nivre, M.-C. de Marneffe, F. Ginter, Y. Goldberg, J. Hajič, C. D. Manning, R. McDonald, S. Petrov, S. Pyysalo, N. Silveira, R. Tsarfaty, and D. Zeman, “Universal Dependencies v1: A multilingual treebank collection,” in Proceedings of the Tenth International Conference on Language Resources and E…
  33. V. Vincze, D. Szauter, A. Almási, G. Móra, Z. Alexin, and J. Csirik, “Hungarian dependency treebank,” in Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10). Valletta, Malta: European Language Resources Association (ELRA), May 2010. [Online]. Available:… find
  34. D. Zeman, M. Popel, M. Straka, J. Hajič, J. Nivre, F. Ginter, J. Luotolahti, S. Pyysalo, S. Petrov, M. Potthast, F. Tyers, E. Badmaeva, M. Gokirmak, A. Nedoluzhko, S. Cinková, J. Hajič jr., J. Hlaváčová, V. Kettnerová, Z. Urešová, J. Kanerva, S. Ojala, A. Missilä, C. D. Manning, S. Schuster, S. R… — CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies
  35. N. Silveira, T. Dozat, M.-C. de Marneffe, S. Bowman, M. Connor, J. Bauer, and C. D. Manning, “A gold standard dependency corpus for English,” in Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014), 2014. find
  36. L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy, “The pile: An 800gb dataset of diverse text for language modeling,” 2020. find
  37. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” 2023. find
  38. S. Longpre, L. Hou, T. Vu, A. Webson, H. W. Chung, Y. Tay, D. Zhou, Q. V. Le, B. Zoph, J. Wei et al., “The flan collection: Designing data and methods for effective instruction tuning,” arXiv preprint arXiv:2301.13688, 2023.
  39. H. Nguyen, “The oig dataset,” Mar 2023. [Online]. Available: https://laion.ai/blog/oig-dataset/ ↗
  40. S. Iyer, X. V. Lin, R. Pasunuru, T. Mihaylov, D. Simig, P. Yu, K. Shuster, T. Wang, Q. Liu, P. S. Koura, X. Li, B. O’Horo, G. Pereyra, J. Wang, C. Dewan, A. Celikyilmaz, L. Zettlemoyer, and V. Stoyanov, “Opt-iml: Scaling language model instruction meta learning through the lens of generalization,” 2…
  41. J. Abadji, P. O. Suarez, L. Romary, and B. Sagot, “Towards a cleaner document-oriented multilingual crawled corpus,” 2022.
  42. L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel, “mT5: A massively multilingual pre-trained text-to-text transformer,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn…
  43. G. Wenzek, M.-A. Lachaux, A. Conneau, V. Chaudhary, F. Guzmán, A. Joulin, and E. Grave, “Ccnet: Extracting high quality monolingual datasets from web crawl data,” 2019.
  44. C. Mou, C. Ha, K. Enevoldsen, and P. Liu, “Chenghaomou/text-dedup: Reference snapshot,” Sep. 2023. [Online]. Available: https://doi.org/10.5281/zenodo.8364980 — [doi:10.5281/zenodo.8364980](https://doi.org/10.5281/zenodo.8364980) ↗
  45. A. Broder, “On the resemblance and containment of documents,” in Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No.97TB100171), 1997, pp. 21–29. find
  46. H. Nomoto, “Interpersonal meaning annotation for asian language corpora: The case of tufs asian language parallel corpus (talpco),” 2019. [Online]. Available: https://api.semanticscholar.org/CorpusID:209441604 ↗
  47. B. Buschbeck-Wolf and M. Exel, “A parallel evaluation data set of software documentation with document structure annotation,” in Workshop on Asian Translation, 2020. [Online]. Available: https://api.semanticscholar.org/CorpusID:221095497 ↗
  48. H. Riza, M. Purwoadi, Gunarso, T. Uliniansyah, A. A. Ti, S. M. Aljunied, L. C. Mai, V. T. Thang, N. P. Thai, V. Chea, R. Sun, S. Sam, S. Seng, K. M. Soe, K. T. Nwet, M. Utiyama, and C. Ding, “Introduction of the asian language treebank,” 2016 Conference of The Oriental Chapter of International Commi… find
  49. F. Ladhak, E. Durmus, C. Cardie, and K. McKeown, “Wikilingua: A new benchmark dataset for multilingual abstractive summarization,” ArXiv, vol. abs/2010.03093, 2020. [Online]. Available: https://api.semanticscholar.org/CorpusID:222177239 ↗
  50. M. Cettolo, C. Girardi, and M. Federico, “WIT3: Web inventory of transcribed and translated talks,” in Proceedings of the 16th Annual conference of the European Association for Machine Translation. Trento, Italy: European Association for Machine Translation, May 28–30 "2012", pp. 261–268. [Online].…
  51. N. team, M. R. Costa-jussà, J. Cross, O. cCelebi, M. Elbayad, K. Heafield, K. Heffernan, E. Kalbassi, J. Lam, D. Licht, J. Maillard, A. Sun, S. Wang, G. Wenzek, A. Youngblood, B. Akula, L. Barrault, G. M. Gonzalez, P. Hansanti, J. Hoffman, S. Jarrett, K. R. Sadagopan, D. Rowe, S. L. Spruit, C. Tran,… find
  52. A. Conneau, G. Lample, R. Rinott, A. Williams, S. R. Bowman, H. Schwenk, and V. Stoyanov, “Xnli: Evaluating cross-lingual sentence representations,” in Conference on Empirical Methods in Natural Language Processing, 2018. [Online]. Available: https://api.semanticscholar.org/CorpusID:52271711 ↗
  53. M. Artetxe, S. Ruder, and D. Yogatama, “On the cross-lingual transferability of monolingual representations,” CoRR, vol. abs/1910.11856, 2019.
  54. K. Viriyayudhakorn and C. Polpanumas, “iapp_wiki_qa_squad,” Feb. 2021. [Online]. Available: https://doi.org/10.5281/zenodo.4539916 — [doi:10.5281/zenodo.4539916](https://doi.org/10.5281/zenodo.4539916) ↗
  55. A. Suriyawongkul, E. Chuangsuwanich, P. “Pythainlp/wisesight-sentiment: First release,” https://doi.org/10.5281/zenodo.3457447 Chormai, and C. Polpanumas, Sep. 2019. [Online]. Available: — [doi:10.5281/zenodo.3457447](https://doi.org/10.5281/zenodo.3457447) ↗
  56. N. Chumpolsathien, “Using knowledge distillation from keyword extraction to improve the informativeness of neural cross-lingual summarization,” Master’s thesis, Beijing Institute of Technology, 2020. find
  57. T. Hasan, A. Bhattacharjee, M. S. Islam, K. Mubasshir, Y.-F. Li, Y.-B. Kang, M. S. Rahman, and R. Shahriyar, “XL-sum: Large-scale multilingual abstractive summarization for 44 languages,” in Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. Online: Association for Computati… find
  58. R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, “Stanford alpaca: An instruction-following llama model,” https://github.com/tatsu-lab/stanford_alpaca, 2023. ↗
  59. M. Conover, M. Hayes, A. Mathur, J. Xie, J. Wan, S. Shah, A. Ghodsi, P. Wendell, M. Zaharia, and R. Xin. (2023) Free dolly: Introducing the world’s first truly open instruction-tuned llm. [Online]. Available: https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction… ↗
  60. B. Guo, X. Zhang, Z. Wang, M. Jiang, J. Nie, Y. Ding, J. Yue, and Y. Wu, “How close is chatgpt to human experts? comparison corpus, evaluation, and detection,” arXiv preprint arxiv:2301.07597, 2023.
  61. A. Kopf, Y. Kilcher, D. von Rutte, S. Anagnostidis, Z. R. Tam, K. Stevens, A. Barhoum, N. M. Duc, O. Stanley, R. Nagyfi, E. Shahul, S. Suri, D. Glushkov, A. Dantuluri, A. Maguire, C. Schuhmann, H. Nguyen, and A. Mattick, “Openassistant conversations - democratizing large language model alignment,” A… — OpenAssistant Conversations — Democratizing Large Language Model Alignment
  62. R. Prabhakar and S. Jairath, “Sambanova sn10 rdu:accelerating software 2.0 with dataflow,” in 2021 IEEE Hot Chips 33 Symposium (HCS), 2021, pp. 1–37. find
  63. L. Gao, J. Tow, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, K. McDonell, N. Muennighoff, J. Phang, L. Reynolds, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou, “A framework for few-shot language model evaluation,” Sep. 2021. [Online]. Available: https://doi.org/10.5281/zenodo.5… — [doi:10.5281/zenodo.5371628](https://doi.org/10.5281/zenodo.5371628) ↗
  64. D. Paperno, G. Kruszewski, A. Lazaridou, Q. N. Pham, R. Bernardi, S. Pezzelle, M. Baroni, G. Boleda, and R. Fernández, “The lambada dataset: Word prediction requiring a broad discourse context,” 2016.
  65. R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi, “Hellaswag: Can a machine really finish your sentence?” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. find
  66. T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal, “Can a suit of armor conduct electricity? a new dataset for open book question answering,” 2018. find
  67. C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova, “Boolq: Exploring the surprising difficulty of natural yes/no questions,” 2019.
  68. P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, “Think you have solved question answering? try arc, the ai2 reasoning challenge,” 2018.
  69. Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi, “Piqa: Reasoning about physical commonsense in natural language,” 2019.
  70. Y. Nie, A. Williams, E. Dinan, M. Bansal, J. Weston, and D. Kiela, “Adversarial nli: A new benchmark for natural language understanding,” 2020.
  71. K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi, “Winogrande: An adversarial winograd schema challenge at scale,” 2019. find
  72. N. Ligeti-Nagy, G. Ferenczi, E. Héja, K. Jelencsik-Mátyus, L. J. Laki, N. Vadász, Z. G. Yang, and T. Vadász, “Hulu: magyar nyelvű benchmark adatbázis kiépítése a neurális nyelvmodellek kiértékelése céljából,” in XVIII. Magyar Számítógépes Nyelvészeti Konferencia, 2022, pp. 431– 446. find
  73. E. M. Ponti, G. G. s, O. Majewska, Q. Liu, I. Vuli’c, and A. Korhonen, “XCOPA: A multilingual dataset for causal commonsense reasoning,” arXiv preprint, 2020. [Online]. Available: https://ducdauge.github.io/files/xcopa.pdf ↗
  74. D. Khashabi, S. Chaturvedi, M. Roth, S. Upadhyay, and D. Roth, “Looking beyond the surface:a challenge set for reading comprehension over multiple sentences,” in Proceedings of North American Chapter of the Association for Computational Linguistics (NAACL), 2018. — Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences
  75. A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “Superglue: A stickier benchmark for general-purpose language understanding systems,” arXiv preprint arXiv:1905.00537, 2019.
  76. T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov, “Natural questions: a benchmark for question answering research,” Transactions of…
  77. S. Zhang, X. Liu, J. Liu, J. Gao, K. Duh, and B. V. Durme, “Record: Bridging the gap between human and machine commonsense reading comprehension,” ArXiv, vol. abs/1810.12885, 2018. [Online]. Available: https://api.semanticscholar.org/CorpusID:53116244 ↗
  78. S. Narayan, S. B. Cohen, and M. Lapata, “Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization,” 2018. — Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/csaki-2023-efficiently-adapting-pretrained-language-models/