Elamur Research⌕
☰
home / papers / 2023 / AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Risks

AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Risks

Paper · 2023 · Alexander Tornede, Difan Deng, Theresa Eimer et al.

Summary

This position paper, which also surveys prior work, argues that Automated Machine Learning (AutoML) and Large Language Models (LLMs) should be developed together rather than separately. It examines the relationship in two directions. First, it considers how AutoML techniques such as Neural Architecture Search (NAS) and Hyperparameter Optimization (HPO) could be applied across the LLM lifecycle (pre-training, fine-tuning and inference) and explains why existing tools are not yet adequate. Second, it explores how LLMs could make AutoML systems easier to interact with, simplify their configuration, and replace some internal components by drawing on meta-knowledge contained in unstructured web text. The paper introduces no new dataset or model. Its contribution is a structured overview of existing work combined with a critical analysis of risks. The authors conclude that research at this intersection is still in its early stages. They identify the concentration of the most capable LLMs in a few mostly private companies as the main obstacle, and they call for an open-source LLM designed specifically for AutoML purposes.

Key points

  • Five obstacles to applying AutoML to LLMs:
    • Pre-training is so expensive that perhaps only one full run is affordable.
    • Lifecycle stages must be tuned one after another rather than jointly.
    • NAS has not yet produced breakthrough architectures.
    • Each stage uses a different metric, and the pre-training metric is only a noisy proxy for downstream performance.
    • The stages mix learning paradigms, while AutoML usually handles one paradigm at a time.
  • Per-stage hyperparameters (Figure 1):
    • Pre-training: data selection, tokenization and architecture.
    • Fine-tuning (task-specific supervised, instruction tuning, alignment): learning rate, batch size and adapter placement.
    • Inference: temperature, top-K, response length, pruning and mixed precision.
  • LLM opportunities for AutoML: The paper discusses better human-machine interaction, easier configuration and replacement of system components. It positions this against established frameworks such as Auto-WEKA, AutoSklearn, AutoGluon and Auto-PyTorch.
  • Risks and proposed mitigations:
    • Risks include weak evaluation, hallucinations causing harmful configurations, excessive user trust in natural-language explanations, and growing resource use.
    • Proposed remedies include an AutoML knowledge graph, safeguards that fall back to default settings, checking LLM claims by actually running experiments, and transparent reporting of resource use in the spirit of Green AutoML.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬German Research Centre for Artificial IntelligenceKaiserslautern · Germany80 works · 29,995 citations🎓University of BolognaBologna · Italy22 works · 2,122 citations🎓Leibniz University HannoverHanover · Germany7 works · 624 citations🔬L3S Research CenterHanover · Germany5 works · 29 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Alexander Tornede, Difan Deng, Theresa Eimer, Joseph Giovanelli, Aditya Mohan, Tim Ruhkopf, Sarah Segel, Daphne Theodorakopoulos, Tanja Tornede, Henning Wachsmuth and Marius Lindauer. 2023. AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Risks. arXiv:2306.08107.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.08107 · arXiv 2306.08107

@misc{tornede2023automl,
  title = {{AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Risks}},
  author = {Alexander Tornede, Difan Deng, Theresa Eimer, Joseph Giovanelli, Aditya Mohan, Tim Ruhkopf, Sarah Segel, Daphne Theodorakopoulos, Tanja Tornede, Henning Wachsmuth and Marius Lindauer},
  year = {2023},
  doi = {10.48550/arxiv.2306.08107},
  eprint = {2306.08107},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.08107}
}

References

206 references, 51 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. E. Adamopoulou and L. Moussiades. Chatbots: History, technology, and applications. Machine Learning with Applications, 2, 2020. find
  2. S. Adriaensen, A. Biedenkapp, G. Shala, N. Awad, T. Eimer, M. Lindauer, and F. Hutter. Automated dynamic algorithm configuration. Journal of Artificial Intelligence Research (JAIR), 75:1633–1699, 2022. find
  3. A. Aghajanyan, L. Zettlemoyer, and S. Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In A. Nenkova and H. T. Ng B. Roark (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Languag…
  4. T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama. Optuna: A next-generation Hyperparameter Optimization framework. In A. Teredesai, V. Kumar, Y. Li, R. Rosales, E. Terzi, and G. Karypis (eds.), Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD’…
  5. I. Alberts, L. Mercolli, T. Pyka, G. Prenosil, K. Shi, A. Rominger, and A. Afshar-Oromieh. Large language models (LLM) and ChatGPT: What will the impact on nuclear medicine be? European Journal of Nuclear Medicine and Molecular Imaging (EJNMMI’23), pp. 1–4, 2023. find
  6. M. Andrychowicz, M. Denil, S. Colmenarejo, M. Hoffman, D. Pfau, T. Schaul, and N. de Freitas. Learning to learn by gradient descent by gradient descent. In D. Lee, M. Sugiyama, U. von Luxburg, I. Guyon, and R. Garnett (eds.), Proceedings of the 30th International Conference on Advances in Neural Inf… find
  7. J. Ba, J. Kiros, and G. Hinton. Layer normalization. arXiv:1607.06450 [stat.ML], 2016.
  8. Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, and o. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv:2204.05862 [cs.LG], 2022. find
  9. M. Balandat, B. Karrer, D. Jiang, S. Daulton, B. Letham, A. Wilson, and E. Bakshy. Botorch: A framework for efficient monte-carlo Bayesian optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Balcan, and H. Lin (eds.), Proceedings of the 34th International Conference on Advances in Neural I… find
  10. R. Balestriero, M. Ibrahim, V. Sobal, A. Morcos, S. Shekhar, T. Goldstein, F. Bordes, A. Bardes, G. Mialon, Y. Tian, A. Schwarzschild, A. Wilson, J. Geiping, Q. Garrido, P. Fernandez, A. Bar, H. Pirsiavash, Y. LeCun, and M. Goldblum. A cookbook of self-supervised learning. arXiv:2304.12210 [cs.LG],… — [arXiv:2304.12210](https://arxiv.org/abs/2304.12210) find
  11. E. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAccT’21), pp. 610–623, 2021. find
  12. G. Bender, P-J. Kindermans, B. Zoph, V. Vasudevan, and Q. Le. Understanding and simplifying one-shot architecture search. In J. Dy and A. Krause (eds.), Proceedings of the 35th International Conference on Machine Learning (ICML’18), volume 80. Proceedings of Machine Learning Research, 2018. find
  13. M. Besta, N. Blach, A. Kubicek, R. Gerstenberger, L. Gianinazzi, J. Gajda, T. Lehmann, M. Podstawski, H. Niewiadomski, P. Nyczyk, and T. Hoefler. Graph of thoughts: Solving elaborate problems with large language models. arXiv:2308.09687, 2023. find
  14. T. De Bie, L. De Raedt, J. Hernández-Orallo, H. Hoos, P. Smyth, and C. Williams. Automating data science. Communications of ACM, 65(3):76–87, 2022. find
  15. B. Bischl, M. Binder, M. Lang, T. Pielok, J. Richter, S. Coors, J. Thomas, T. Ullmann, M. Becker, A.-L. Boulesteix, D. Deng, and M. Lindauer. Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discover… find
  16. O. Bohdal, L. Balles, M. Wistuba, B. Ermis, C. Archambeau, and G. Zappella. PASHA: Efficient HPO and NAS with progressive resource allocation. In International Conference on Learning Representations (ICLR’23), 2023. Published online: iclr.cc. find
  17. A. Brock, T. Lim, J. Ritchie, and N. Weston. Smash: One-shot model architecture search through hypernetworks. In Proceedings of the International Conference on Learning Representations (ICLR’18), 2018. Published online: iclr.cc. find
  18. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S… find
  19. Curran Associates, 2020. find
  20. G. Casalicchio, J. Bossek, M. Lang, D. Kirchhoff, P. Kerschke, B. Hofner, H. Seibold, J. Vanschoren, and B. Bischl. OpenML: An R package to connect to the machine learning platform OpenML. Computational Statistics, 34(3):977–991, 2019. find
  21. A. Chen, D. Dohan, and D. So. EvoPrompting: Language models for code-level neural architecture search. arXiv:2302.14838 [cs.NE], 2023. — [arXiv:2302.14838](https://arxiv.org/abs/2302.14838) find
  22. D. Chen, Y. Li, M. Qiu, Z. Z. Wang, B. Li, B. Ding, H. Deng, J. Huang, W. Lin, and J. Zhou. AdaBERT: Task-adaptive BERT compression with differentiable neural architecture search. In C. Bessiere (ed.), Proceedings of the 29th International Joint Conference on Artificial Intelligence (IJCAI’20), 2020… find
  23. M. Chen, H. Peng, J. Fu, and H. Ling. Autoformer: Searching transformers for visual recognition. In Proceedings of the 24nd IEEE/CVF International Conference on Computer Vision (ICCV’21), pp. 12270– 12280. cvfandieee, IEEE, 2021. find
  24. X. Chen, L. Xie, J. Wu, and Q. Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of the 22nd IEEE/CVF International Conference on Computer Vision (ICCV’19), pp. 1294–1303. cvfandieee, IEEE, 2019.
  25. Y. Chen, X. Song, C. Lee, Z. Wang, Q. Zhang, D. Dohan, K. Kawakami, G. Kochanski, A. Doucet, M. Ranzato, S. Perel, and N. de Freitas. Towards learning universal hyperparameter optimizers with transformers. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Proceedings of th… find
  26. K. Chitty-Venkata, M. Emani, V. Vishwanath, and A. Somani. Neural architecture search for transformers: A survey. IEEE Access, 2022. find
  27. H. Choi, J. Moran, N. Matsumoto, M. Hernandez, and J. Moore. Aliro: an automated machine learning tool leveraging large language models. Bioinformatics, 2023. find
  28. A. Dakhel, V. Majdinasab, A. Nikanjam, F. Khomh, M. Desmarais, and Z. Jiang. GitHub copilot AI pair programmer: Asset or liability? Journal of Systems and Software, pp. 111734, 2023. find
  29. M. Deb, B. Deiseroth, S. Weinbach, P. Schramowski, and K. Kersting. AtMan: Understanding transformer predictions through memory efficient attention manipulation. arXiv:2301.08110 [cs.LG], 2023. find
  30. M. Dehghani, Y. Tay, A. Arnab, L. Beyer, and A. Vaswani. The efficiency misnomer. In Proceedings of the International Conference on Learning Representations (ICLR’22), 2022. Published online: iclr.cc. find
  31. D. Deng, F. Karl, F. Hutter, B. Bischl, and M. Lindauer. Efficient automated deep learning for time series forecasting. In Proceedings of the European Conference on Machine Learning and Knowledge Discovery in Databases (ECML/PKDD’22), 2022. find
  32. T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer. LLM.int8(): 8-bit matrix multiplication for transformers at scale. arXiv:2208.07339 [cs.LG], 2022. find
  33. J. Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, and T. Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human L…
  34. X. Dong and Y. Yang. NAS-Bench-201: Extending the scope of reproducible Neural Architecture Search. In Proceedings of the International Conference on Learning Representations (ICLR’20), 2020. Published online: iclr.cc. find
  35. T. Eimer, M. Lindauer, and R. Raileanu. Hyperparameters in reinforcement learning and how to tune them. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning (ICML’23), volume 202 of Proceedings of M… find
  36. T. Elsken, J. Metzen, and F. Hutter. Neural Architecture Search: A survey. Journal of Machine Learning Research, 20(55):1–21, 2019. find
  37. N. Erickson, J. Mueller, A. Shirkov, H. Zhang, P. Larroy, M. Li, and A. Smola. Autogluon-tabular: Robust and accurate automl for structured data. arXiv:2003.06505 [stat.ML], 2020. — [arXiv:2003.06505](https://arxiv.org/abs/2003.06505) find
  38. G. Faggioli, L. Dietz, C. Clarke, G. Demartini, M. Hagen, C. Hauff, N. Kando, E. Kanoulas, M. Potthast, B. Stein, and H. Wachsmuth. Perspectives on large language models for relevance judgment, 2023. find
  39. P. Fernandes, A. Madaan, E. Liu, A. Farinhas, P. Martins, A. Bertsch, J. de Souza, S. Zhou, T. Wu, G. Neubig, and A. Martins. Bridging the gap: A survey on integrating (Human) feedback for natural language generation. arXiv:2305.00955 [cs.CL], 2023.
  40. M. Feurer and F. Hutter. Towards further automation in AutoML. In R. Garnett, F. Hutter, J. Vanschoren, P. Brazdil, R. Caruana, C. Giraud-Carrier, I. Guyon, and B. Kégl (eds.), ICML workshop on Automated Machine Learning (AutoML workshop 2018), 2018. find
  41. M. Feurer and F. Hutter. Hyperparameter Optimization. In Hutter et al. (2019), chapter 1, pp. 3 – 38. Available for free at http://automl.org/book. ↗
  42. M. Feurer, A. Klein, K. Eggensperger, J. Springenberg, M. Blum, and F. Hutter. Efficient and robust automated machine learning. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Proceedings of the 29th International Conference on Advances in Neural Information Processing Systems… find
  43. M. Feurer, J. Springenberg, and F. Hutter. Initializing Bayesian Hyperparameter Optimization via meta-learning. In B. Bonet and S. Koenig (eds.), Proceedings of the Twenty-ninth AAAI Conference on Artificial Intelligence (AAAI’15), pp. 1128–1135. AAAI Press, 2015b. find
  44. M. Feurer, A. Klein, K. Eggensperger, J. Springenberg, M. Blum, and F. Hutter. Auto-sklearn: Efficient and robust automated machine learning. In Hutter et al. (2019), chapter 6, pp. 113–134. Available for free at http://automl.org/book. ↗
  45. M. Feurer, K. Eggensperger, S. Falkner, M. Lindauer, and F. Hutter. Auto-Sklearn 2.0: Hands-free automl via meta-learning. Journal of Machine Learning Research, 23(261):1–61, 2022. find
  46. L. Franceschi, M. Donini, P. Frasconi, and M. Pontil. Forward and Reverse Gradient-Based Hyperparameter Optimization. In D. Precup and Y. Teh (eds.), Proceedings of the 34th International Conference on Machine Learning (ICML’17), volume 70, pp. 1165–1173. Proceedings of Machine Learning Research, 20… find
  47. N. Friedman. Introducing GitHub copilot: Your AI pair programmer. GitHub Blog, 2021. find
  48. J. Gao, H. Xu, H. Shi, X. Ren, P. Yu, X. Liang, X. Jiang, and Z. Li. AutoBERT-zero: Evolving BERT backbone from scratch. In Q. Yang, K. Leyton-Brown, and Mausam (eds.), Proceedings of the Thirty-Fifth Conference on Artificial Intelligence (AAAI’21). Association for the Advancement of Artificial Inte… find
  49. J. Geiping and T. Goldstein. Cramming: Training a language model on a single gpu in one day. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning (ICML’23), volume 202 of Proceedings of Machine Lear… find
  50. L. Gilpin, D. Bau, B. Yuan, A. Bajwa, M. Specter, and L. Kagal. Explaining explanations: An overview of interpretability of machine learning. In Proceedings of the International Conference on Data Science and Advanced Analytics (DSAA’18), pp. 80–89, 2018. find
  51. M. Gira, R. Zhang, and K. Lee. Debiasing pre-trained language models via efficient fine-tuning. In Proceedings of Workshop on Language Technology for Equality, Diversity and Inclusion (LT-EDI’22), pp. 59–69, 2022. find
  52. A. Glaese, N. McAleese, M. Trebacz, J. Aslanides, V. Firoiu, T. Ewalds, M. Rauh, L. Weidinger, M. Chadwick, P. Thacker, L. Campbell-Gillingham, J. Uesato, P. Huang, R. Comanescu, F. Yang, A. See, S. Dathathri, R. Greig, C. Chen, D. Fritz, J. Sanchez Elias, R. Green, S. Mokrá, N. Fernando, B. Wu, R.… — Improving alignment of dialogue agents via targeted human judgements
  53. V. Godbole, G. Dahl, J. Gilmer, C. Shallue, and Z. Nado. Deep learning tuning playbook, 2023. Version 1. find
  54. P. Grimal, H. L. Borgne, O. Ferret, and J. Tourille. TIAM – a metric for evaluating alignment in text-to-image generation. arXiv:2307.05134 [cs.CV], 2023. — [arXiv:2307.05134](https://arxiv.org/abs/2307.05134) find
  55. Y. Gu, L. Dong, F. Wei, and M. Huang. Knowledge distillation of large language models. arXiv:2306.08543, 2023.
  56. T. Han, L. Adams, J. Papaioannou, P. Grundmann, T. Oberhauser, A. Löser, D. Truhn, and K. Bressem. MedAlpaca - an open-source collection of medical conversational AI models and training data. arXiv:2304.08247 [cs.CL], 2023.
  57. D. Hendrycks and K. Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. arXiv:1606.08415 [cs.LG], 2016. find
  58. E. Hernandez, B. Li, and J. Andreas. Measuring and manipulating knowledge representations in language models. arXiv:2305.02499 [cs.CL], 2023.
  59. A. Hogan, E. Blomqvist, M. Cochez, C. d’Amato, G. de Melo, C. Gutierrez, S. Kirrane, J. Labra Gayo, R. Navigli, S. Neumaier, A. Ngonga Ngomo, A. Polleres, S. Rashid, A. Rula, L. Schmelzeisen, J. Sequeda, S. Staab, and A. Zimmermann. Knowledge graphs. ACM Computing Surveys, 54(4):71:1–71:37, 2022.
  60. N. Hollmann, S. Müller, K. Eggensperger, and F. Hutter. TabPFN: A transformer that solves small tabular classification problems in a second. In International Conference on Learning Representations (ICLR’23), 2023a. Published online: iclr.cc. find
  61. N. Hollmann, S. Müller, and F. Hutter. GPT for semi-automated data science: Introducing CAAFE for context-aware automated feature engineering. arXiv:2305.03403 [cs.AI], 2023b. — [arXiv:2305.03403](https://arxiv.org/abs/2305.03403) find
  62. N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. de Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly. Parameter-efficient transfer learning for NLP. In K. Chaudhuri and R. Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning (ICML’19), volume 97. Proce… find
  63. C. Hsieh, C. Li, C. Yeh, H. Nakhost, Y. Fujii, A. Ratner, R. Krishna, C. Lee, and T. Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In A. Rogers, J. L. Boyd-Graber, and N. Okazaki (eds.), Proceedings of the Association for Comp…
  64. E. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In Proceedings of the International Conference on Learning Representations (ICLR’22), 2022a. Published online: iclr.cc.
  65. Y. Hu, Y. Liang, Z. Guo, R. Wan, X. Zhang, Y. Wei, Q. Gu, and J. Sun. Angle-based search space shrinking for neural architecture search. In A. Vedaldi, H. Bischof, T. Brox, and J. Frahm (eds.), 16th European Conference on Computer Vision (ECCV’20), pp. 119–134. Springer, 2020. find
  66. Y. Hu, X. Wang, and Q. Gu. PWSNAS: Powering weight sharing NAS with general search space shrinking framework. IEEE transactions on Neural Networks and Learning Systems, 2022b. find
  67. F. Hutter. Deep learning 2.0: Extending the power of deep learning to the meta-level. AutoML.org Blog, 2022. find
  68. F. Hutter, L. Kotthoff, and J. Vanschoren (eds.). Automated Machine Learning: Methods, Systems, Challenges. Springer, 2019. Available for free at http://automl.org/book. ↗
  69. C. Hvarfner, D. Stoll, A. Souza, L. Nardi, M. Lindauer, and F. Hutter. πBO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization. In Proceedings of the International Conference on Learning Representations (ICLR’22), 2022. Published online: iclr.cc. find
  70. S. Jawed, H. Jomaa, L. Thieme, and J. Grabocka. Multi-task learning curve forecasting across hyperparameter configurations and datasets. In N. Oliver, F. Pérez-Cruz, S. Kramer, J. Read, and J. A. Lozano (eds.), Proceedings of the 22th European Conference on Machine Learning and Principles and Practi… find
  71. Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. Bang, A. Madotto, and P. Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, pp. 248:1–248:38, 2023.
  72. M. Jiang, T. Rocktäschel, and E. Grefenstette. arXiv:2211.07819 [cs.AI], 2023. — [arXiv:2211.07819](https://arxiv.org/abs/2211.07819) find
  73. General intelligence requires rethinking exploration. H. Jin, Q. Song, and X. Hu. Auto-Keras: An efficient neural architecture search system. In A. Teredesai, V. Kumar, Y. Li, R. Rosales, E. Terzi, and G. Karypis (eds.), Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discov… find
  74. J. Kaplan, S. McCandlish, T. Henighan, T. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models, 2020.
  75. F. Karl, T. Pielok, J. Moosbauer, F. Pfisterer, S. Coors, M. Binder, L. Schneider, J. Thomas, J. Richter, M. Lang, E. Garrido-Merchán, J. Branke, and B. Bischl. Multi-objective hyperparameter optimization – an overview. Transactions of Evolutionary Learning and Optimization, 3(4):1––50, 2023. find
  76. E. Kasneci, K. Seßler, S. Küchemann, M. Bannert, D. Dementieva, F. Fischer, U. Gasser, G. Groh, S. Günnemann, E. Hüllermeier, S. Krusche, G. Kutyniok, T. Michaeli, C. Nerdel, J. Pfeffer, O. Poquet, M. Sailer, A. Schmidt, T. Seidel, M. Stadler, J. Weller, J. Kuhn, and G. Kasneci. ChatGPT for good? on… — ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education
  77. D. Kingma and J. Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR’15), 2015. Published online: iclr.cc.
  78. L. Kirsch, J. Harrison, J. Sohl-Dickstein, and L. Metz. General-purpose in-context learning by meta-learning transformers. arXiv:2212.04458 [cs.LG], abs/2212.04458, 2022. — [arXiv:2212.04458](https://arxiv.org/abs/2212.04458) find
  79. A. Klein, S. Falkner, J. Springenberg, and F. Hutter. Learning curve prediction with Bayesian neural networks. In Proceedings of the International Conference on Learning Representations (ICLR’17), 2017. Published online: iclr.cc. find
  80. N. Klyuchnikov, I. Trofimov, E. Artemova, M. Salnikov, M. Fedorov, A. Filippov, and E. Burnaev. Nas-bench-nlp: Neural architecture search benchmark for natural language processing. IEEE Access, 10:45736–45747, 2022. find
  81. J. Kok. On Data Snooping and Multiple Outlier Testing. US Department of Commerce, National Oceanic and Atmospheric Administration, National Ocean Service, Charting and Geodetic Services, 1984. find
  82. B. Komer, J. Bergstra, and C. Eliasmith. Hyperopt-sklearn: Automatic hyperparameter configuration for scikit-learn. In F. Hutter, R. Caruana, R. Bardenet, M. Bilenko, I. Guyon, B. Kégl, and H. Larochelle (eds.), ICML workshop on Automated Machine Learning (AutoML workshop 2014), 2014. find
  83. A. Köpf, Y. Kilcher, D. von Rütte, S. Anagnostidis, Z. Tam, K. Stevens, A. Barhoum, N. Duc, O. Stanley, R. Nagyfi, S. ES, S. Suri, D. Glushkov, A. Dantuluri, A. Maguire, C. Schuhmann, H. Nguyen, and A. Mattick. OpenAssistant conversations - democratizing large language model alignment. arXiv:2304.07… — OpenAssistant Conversations — Democratizing Large Language Model Alignment
  84. L. Kotthoff, C. Thornton, H. Hoos, F. Hutter, and K. Leyton-Brown. Auto-weka 2.0: Automatic model selection and hyperparameter optimization in WEKA. J. Mach. Learn. Res., 18:25:1–25:5, 2017. find
  85. L. Kotthoff, C. Thornton, H. Hoos, F. Hutter, and K. Leyton-Brown. Auto-WEKA: automatic model selection and hyperparameter optimization in WEKA. In Hutter et al. (2019), chapter 4, pp. 81–95. Available for free at http://automl.org/book. ↗
  86. T. Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Conference of the North American Chapter of the Association for Computational Linguistics (ACL’18), pp. 66–75, 2018.
  87. S. Kumar, V. Balachandran, L. Njoo, A. Anastasopoulos, and Y. Tsvetkov. Language generation models can cause harm: So what can we do about it? an actionable survey. In Proceedings of the 17th Conference of the European American Chapter of the Association for Computational Linguistics: Human Language…
  88. P. Kung, F. Yin, D. Wu, K. Chang, and N. Peng. Active instruction tuning: Improving cross-task generalization by training on prompt sensitive tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP’23), 2023. find
  89. N. Lambert, L. Castricato, L. von Werra, and A. Havrilla. Illustrating reinforcement learning from human feedback (RLHF). Hugging Face Blog, 2022. https://huggingface.co/blog/rlhf. ↗
  90. Q. Lan, A. Mahmood, S. Yan, and Z. Xu. Learning to optimize for reinforcement learning. arXiv:2302.01470, 2023. — [arXiv:2302.01470](https://arxiv.org/abs/2302.01470) find
  91. R. Laroche and R. Feraud. Reinforcement learning algorithm selection. In Proceedings of the International Conference on Learning Representations (ICLR’18), 2018. Published online: iclr.cc. find
  92. H. Law, P. Zhao, L. Chan, J. Huang, and D. Sejdinovic. Hyperparameter learning via distributional transfer. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 6804–6815. Curran Associates, Inc., 2019. find
  93. J. Lee, R. Tang, and J. Lin. What would elsa do? arXiv:1911.03090 [cs.CL], 2019. freezing layers during transformer fine-tuning. — [arXiv:1911.03090](https://arxiv.org/abs/1911.03090) find
  94. M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 57th Conference of the North American Chapter of the Associatio…
  95. A. Li, O. Spyra, S. Perel, V. Dalibard, M. Jaderberg, C. Gu, D. Budden, T. Harley, and P. Gupta. A generalized framework for population based training. In A. Teredesai, V. Kumar, Y. Li, R. Rosales, E. Terzi, and G. Karypis (eds.), Proceedings of the 25th ACM SIGKDD International Conference on Knowle… find
  96. J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv:2301.12597 [cs.CV], 2023.
  97. L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar. Hyperband: A novel bandit-based approach to Hyperparameter Optimization. Journal of Machine Learning Research, 18(185):1–52, 2018. find
  98. Y. Li, Y. Shen, H. Jiang, W. Zhang, Z. Yang, C. Zhang, and B. Cui. Transbo: Hyperparameter optimization via two-phase transfer learning. In A. Zhang and H. Rangwala (eds.), Proceedings of the 28th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’22). ACM Press, 2022. find
  99. P. Liang, I. Li, E. Zheng, Y. Lim, R. Salakhutdinov, and L. Morency. Towards debiasing sentence representations. In Proceedings of the 57th Conference of the North American Chapter of the Association for Computational Linguistics (ACL’20), pp. 5502–5515, 2020.
  100. P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. Cosgrove, C. D. Manning, C. Ré, C. Acosta-Navas, D. Hudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong, H. Ren, H. Yao, J. Wang, K. Santhanam, L. J. Or… find
  101. M. Lindauer and F. Hutter. Best practices for scientific research on Neural Architecture Search. Journal of Machine Learning Research, 21(243):1–18, 2020. find
  102. M. Lindauer and A. Tornede. Rethinking AutoML: Advancing from a machine-centered to human-centered paradigm. AutoML.org Blog, 2022. find
  103. M. Lindauer, K. Eggensperger, M. Feurer, A. Biedenkapp, D. Deng, C. Benjamins, T. Ruhkopf, R. Sass, and F. Hutter. SMAC3: A versatile bayesian optimization package for Hyperparameter Optimization. Journal of Machine Learning Research, 23(54):1–9, 2022. find
  104. B. Liu, C. Chen, C. Liao, Z. Gong, H. Wang, Z. Lei, M. Liang, D. Chen, M. Shen, H. Zhou, H. Yu, and J. Li. Mftcoder: Boosting code llms with multitask fine-tuning. arXiv:2311.02303 [cs.LG], 2023. — [arXiv:2311.02303](https://arxiv.org/abs/2311.02303) find
  105. H. Liu, K. Simonyan, and Y. Yang. DARTS: Differentiable architecture search. In Proceedings of the International Conference on Learning Representations (ICLR’19), 2019a. Published online: iclr.cc. find
  106. T. Liu, N. Astorga, N. Seedat, and M. van der Schaar. Large language models to enhance bayesian optimization. In International Conference on Learning Representations (ICLR’24), 2024. Published online: iclr.cc. find
  107. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv:1907.11692 [cs.CL], 2019b.
  108. J. Lorraine, P. Vicol, and D. Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In S. Chiappa and R. Calandra (eds.), Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS’20), volume 108, pp. 1540–1552. Proceedings of Machine… find
  109. I. Loshchilov and F. Hutter. SGDR: Stochastic gradient descent with warm restarts. In Proceedings of the International Conference on Learning Representations (ICLR’17), 2017. Published online: iclr.cc. find
  110. I. Loshchilov and F. Hutter. Fixing weight decay regularization in adam. In Proceedings of the International Conference on Learning Representations (ICLR’18), 2018. Published online: iclr.cc. find
  111. J. Luketina, M. Berglund, K. Greff, and T. Raiko. Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters. In M. Balcan and K. Weinberger (eds.), Proceedings of the 33rd International Conference on Machine Learning (ICML’17), volume 48, pp. 2952–2960. Proceedings of Machine Learn… find
  112. F. Mohr M. Wever and E. Hüllermeier. ML-Plan for unlimited-length machine learning pipelines. In R. Garnett, F. Hutter, J. Vanschoren, P. Brazdil, R. Caruana, C. Giraud-Carrier, I. Guyon, and B. Kégl (eds.), ICML workshop on Automated Machine Learning (AutoML workshop 2018), 2018. find
  113. M. MacKay, P. Vicol, J. Lorraine, D. Duvenaud, and R. Grosse. Self-tuning networks: Bilevel optimization of hyperparameters using structured best-response functions. In Proceedings of the International Conference on Learning Representations (ICLR’19), 2019. Published online: iclr.cc. find
  114. D. Maclaurin, D. Duvenaud, and R. Adams. Gradient-based Hyperparameter Optimization through Reversible Learning. In F. Bach and D. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning (ICML’15), volume 37, pp. 2113–2122. Omnipress, 2015. find
  115. R. Mahabadi, S. Ruder, M. Dehghani, and J. Henderson. Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks. In A. Nenkova and H. T. Ng B. Roark (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: H…
  116. A. Makarova, H. Shen, V. Perrone, A. Klein, J.B. Faddoul, A. Krause, M. Seeger, and C. Archambeau. Automatic termination for hyperparameter optimization. In I. Guyon, M. Lindauer, M. van der Schaar, F. Hutter, and R. Garnett (eds.), Proceedings of the First International Conference on Automated Mach… find
  117. N. Mallik, C. Hvarfner, D. Stoll, M. Janowski, E. Bergman, M. Lindauer, L. Nardi, and F. Hutter. PriorBand: HyperBand + human expert knowledge. In F. Ferreira, Q. Lei, E. Triantafillou, J. Vanschoren, and H. Yao (eds.), NeurIPS 2022 Workshop on Meta-Learning, 2022. find
  118. N. Mallik, C. Hvarfner, E. Bergman, D. Stoll, M. Janowski, M. Lindauer, L. Nardi, and F. Hutter. PriorBand: Practical hyperparameter optimization in the age of deep learning. In Proceedings of the 37th International Conference on Advances in Neural Information Processing Systems (NeurIPS’23). Curran… find
  119. Y. Mehta, C. White, A. Zela, A. Krishnakumar, G. Zabergja, S. Moradian, M. Safari, K. Yu, and F. Hutter. NAS-Bench-Suite: NAS evaluation is (now) surprisingly easy. In Proceedings of the International Conference on Learning Representations (ICLR’22), 2022. Published online: iclr.cc. find
  120. V. Mnih, A. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In M. Balcan and K. Weinberger (eds.), Proceedings of the 33rd International Conference on Machine Learning (ICML’17), volume 48, pp. 1928–1937. Proce… find
  121. A. Mohan, C. Benjamins, K. Wienecke, A. Dockhorn, and M. Lindauer. Autorl hyperparameter landscapes. In A. Faust, C. White, F. Hutter, R. Garnett, and J. Gardner (eds.), Proceedings of the Second International Conference on Automated Machine Learning. Proceedings of Machine Learning Research, 2023. find
  122. F. Mohr, M. Wever, A. Tornede, and E. Hüllermeier. Predicting machine learning pipeline runtimes in the context of automated machine learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(9):3055–3066, 2021. find
  123. J. Moosbauer, J. Herbinger, G. Casalicchio, M. Lindauer, and B. Bischl. Explaining hyperparameter optimization via partial dependence plots. In M. Ranzato, A. Beygelzimer, K. Nguyen, P. Liang, J. Vaughan, and Y. Dauphin (eds.), Proceedings of the 35th International Conference on Advances in Neural I… find
  124. J. Moosbauer, M. Binder, L. Schneider, F. Pfisterer, M. Becker, M. Lang, L. Kotthoff, and B. Bischl. Automated benchmark-driven design and explanation of hyperparameter optimizers. Evolutionary Computation, 26(6): 1336–1350, 2022a. find
  125. J. Moosbauer, G. Casalicchio, M. Lindauer, and B. Bischl. Enhancing explainability of hyperparameter optimization via bayesian algorithm execution. arXiv:2206.05447 [cs.LG], 2022b. — [arXiv:2206.05447](https://arxiv.org/abs/2206.05447) find
  126. A. Morales-Hernández, I. Van Nieuwenhuyse, and S. Gonzalez. A survey on multi-objective hyperparameter optimization algorithms for machine learning. arXiv:2111.13755 [cs.LG], 2021. — [arXiv:2111.13755](https://arxiv.org/abs/2111.13755) find
  127. S. Müller, M. Feurer, N. Hollmann, and F. Hutter. Pfns4bo: In-context learning for bayesian optimization. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning (ICML’23), volume 202 of Proceedings of… find
  128. M. Nasir, S. Earle, J. Togelius, S. James, and C. Cleghorn. Llmatic: Neural architecture search via large language models and quality-diversity optimization. arXiv:2306.01102 [cs.NE], 2023. — [arXiv:2306.01102](https://arxiv.org/abs/2306.01102) find
  129. V. Nguyen, S. Gupta, S. Rana, C. Li, and S. Venkatesh. Filtering bayesian optimization approach in weakly specified search space. Knowledge and Information Systems (KAIS’19), 60(1):385–413, 2019. find
  130. X. Ning, Z. Lin, Z. Zhou, H. Yang, and Y. Wang. Skeleton-of-thought: Large language models can do parallel decoding. arXiv:2307.15337 [cs.CL], 2023.
  131. N. Noonan. Creative mutation: A prescriptive approach to the use of ChatGPT and large language models in lawyering. Social Science Research Network (SSRN 4406907), 2023. find
  132. R. Olson and J. Moore. TPOT: A tree-based pipeline optimization tool for automating machine learning. In Hutter et al. (2019), pp. 151–160. Available for free at http://automl.org/book. ↗
  133. OpenAI. Introducing ChatGPT. OpenAI Blog, 2022. find
  134. OpenAI. Gpt-4 is OpenAI’s most advanced system, producing safer and more useful responses. OpenAI Blog, 2023. find
  135. J. Parker-Holder, V. Nguyen, and S. J. Roberts. Provably efficient online Hyperparameter Optimization with population-based bandits. In H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Balcan, and H. Lin (eds.), Proceedings of the 34th International Conference on Advances in Neural Information Processin… find
  136. J. Parker-Holder, R. Rajan, X. Song, A. Biedenkapp, Y. Miao, T. Eimer, B. Zhang, V. Nguyen, R. Calandra, A. Faust, F. Hutter, and M. Lindauer. Automated reinforcement learning (AutoRL): A survey and open problems. Journal of Artificial Intelligence Research (JAIR), 74:517–568, 2022. find
  137. V. Perrone, R. Jenatton, M. Seeger, and C. Archambeau. Scalable hyperparameter transfer learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Proceedings of the 31st International Conference on Advances in Neural Information Processing Systems (NeurIP… find
  138. F. Pfisterer, J. Thomas, and B. Bischl. Towards human centered AutoML. arXiv:1911.02391 [cs.HC], 2019. — [arXiv:1911.02391](https://arxiv.org/abs/1911.02391) find
  139. X. Qi, Y. Zeng, T. Xie, P. Chen, R. Jia, P. Mittal, and P. Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv:2310.03693 [cs.CL], 2023.
  140. A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. Improving language understanding by generative pre-training. OpenAI Blog, 2018. find
  141. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9, 2019. find
  142. R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. Manning, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv:2305.18290 [cs.LG], 2023.
  143. T. Ruhkopf, A. Mohan, D. Deng, A. Tornede, F. Hutter, and M. Lindauer. MASIF: Meta-learned algorithm selection using implicit fidelity information. Transactions on Machine Learning Research, 2023. find
  144. A. Saravanan, D. Mullick, H. Rahman, and N. Hegde. FineDeb: A debiasing framework for language models. arXiv:2305.02499 [cs.CL], 2023.
  145. E. Schede, J. Brandt, A. Tornede, M. Wever, V. Bengs, E. Hüllermeier, and K. Tierney. A survey of methods for automated algorithm configuration. jair, 75:425–487, 2022. find
  146. T. Schick, S. Udupa, and H. Schütze. Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in NLP. Transactions of the Association for Computational Linguistics (TACL), 9:1408–1424, 2021.
  147. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv:1707.06347 [cs.LG], 2017. find
  148. M. Schuster and K. Nakajima. Japanese and korean coice search. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP’12), pp. 5149–5152, 2012. find
  149. S. Segel, H. Graf, A. Tornede, B. Bischl, and M. Lindauer. Symbolic explanations for hyperparameter optimization. In A. Faust, C. White, F. Hutter, R. Garnett, and J. Gardner (eds.), Proceedings of the Second International Conference on Automated Machine Learning. Proceedings of Machine Learning Res… find
  150. R. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Conference of the North American Chapter of the Association for Computational Linguistics (ACL’16), pp. 1715–1725, 2016.
  151. N. Shazeer. GLU variants improve transformer. arXiv:2002.05202 [cs.LG], 2020. find
  152. N. Shazeer and M. Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In J. Dy and A. Krause (eds.), Proceedings of the 35th International Conference on Machine Learning (ICML’18), volume 80, pp. 4603–4611. Proceedings of Machine Learning Research, 2018. find
  153. S. Shen, Z. Dong, J. Ye, L. Ma, Z. Yao, A. Gholami, M. Mahoney, and K. Keutzer. Q-BERT: hessian based ultra low precision quantization of BERT. In F. Rossi, V. Conitzer, and F. Sha (eds.), Proceedings of the Thirty-Fourth Conference on Artificial Intelligence (AAAI’20). Association for the Advanceme… find
  154. H. Shi, R. Pi, H. Xu, Z. Li, J. Kwok, and T. Zhang. Bridging the gap between sample-based and one-shot Neural Architecture Search with BONAS. In H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Balcan, and H. Lin (eds.), Proceedings of the 34th International Conference on Advances in Neural Information… find
  155. T. Shin, Y. Razeghi, R. Logan IV, E. Wallace, and S. Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNL…
  156. M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv:1909.08053 [cs.CL], 2019. find
  157. D. So, W. Mańke, H. Liu, Z. Dai, N. Shazeer, and Q. Le. Searching for efficient transformers for language modeling. In M. Ranzato, A. Beygelzimer, K. Nguyen, P. Liang, J. Vaughan, and Y. Dauphin (eds.), Proceedings of the 35th International Conference on Advances in Neural Information Processing Sys… find
  158. Xingyou Song, Sagi Perel, Chansoo Lee, Greg Kochanski, and Daniel Golovin. Open source vizier: distributed infrastructure and api for reliable and flexible blackbox optimization. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato (eds.), Proceedings of the 39th International… find
  159. T. Sorensen, J. Robinson, C. Rytting, A. Shaw, K. Rogers, A. Delorey, M. Khalil, N. Fulda, and D. Wingate. An information-theoretic approach to prompt engineering without ground truth labels. In Proceedings of the 59th Conference of the North American Chapter of the Association for Computational Lin…
  160. A. Souza, L. Nardi, L. Oliveira, K. Olukotun, M. Lindauer, and F. Hutter. Bayesian optimization with a prior for the optimum. In N. Oliver, F. Pérez-Cruz, S. Kramer, J. Read, and J. Lozano (eds.), Machine Learning and Knowledge Discovery in Databases (ECML/PKDD’21), volume 12975 of Lecture Notes in… find
  161. J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, and Y. Liu. RoFormer: Enhanced transformer with rotary position embedding. arXiv:2104.09864 [cs.CL], 2022.
  162. R. Thoppilan, D. De Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H. Cheng, A. Jin, T. Bos, L. Baker, Y. Du, Y. Li, H. Lee, H. Zheng, A. Ghafouri, M. Menegali, Y. Huang, M. Krikun, D. Lepikhin, J. Qin, D. Chen, Y. Xu, Z. Chen, A. Roberts, M. Bosma, Y. Zhou, C. Chang, I. Krivokon, W. Rusch, M. Picke… — LaMDA: Language Models for Dialog Applications
  163. C. Thornton, F. Hutter, H. Hoos, and K. Leyton-Brown. Auto-WEKA: combined selection and Hyperparameter Optimization of classification algorithms. In I. Dhillon, Y. Koren, R. Ghani, T. Senator, P. Bradley, R. Parekh, J. He, R. Grossman, and R. Uthurusamy (eds.), The 19th ACM SIGKDD International Conf… find
  164. A. Tornede, L. Gehring, T. Tornede, M. Wever, and E. Hüllermeier. Algorithm selection on a meta level. Machine Learning, 2022. find
  165. T. Tornede, A. Tornede, J. Hanselle, F. Mohr, M. Wever, and E. Hüllermeier. Towards green automated machine learning: Status quo and future directions. Journal of Artificial Intelligence Research, 77:427–457, 2023. find
  166. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. LLaMA: Open and efficient foundation language models. arXiv:2302.13971 [cs.CL], 2023.
  167. Y. Tsai, Y. Tsai, B. Huang, C. Yang, and S. Lin. arXiv:2309.01125 [cs.LG], 2023. — [arXiv:2309.01125](https://arxiv.org/abs/2309.01125) find
  168. Automl-gpt: Large language model for automl. K. van der Blom, A. Serban, H. Hoos, and J. Visser. AutoML adoption in ML software. In K. Eggensperger, F. Hutter, E. LeDell, M. Lindauer, G. Shala, J. Vanschoren, and C. White (eds.), ICML workshop on Automated Machine Learning (AutoML workshop 2021), 20… find
  169. J. Vanschoren. Meta-learning. In Hutter et al. (2019), chapter 2, pp. 35–61. Available for free at http://automl.org/book. ↗
  170. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Proceedings of the 31st International Conference on Advances in Neural…
  171. H. Wachsmuth, M. Rose, and G. Engels. Automatic pipeline construction for real-time annotation. In A. Gelbukh (ed.), Proceedings of the International Conference on Computational Linguistics and Intelligent Text Processing (CICLing’13), volume 7816, pp. 38–49, 2013. find
  172. X. Wan, C. Lu, J. Parker-Holder, P. Ball, V. Nguyen, B. Ru, and M. Osborne. Bayesian generational population-based training. In I. Guyon, M. Lindauer, M. van der Schaar, F. Hutter, and R. Garnett (eds.), Proceedings of the First International Conference on Automated Machine Learning. Proceedings of… find
  173. C. Wang, Q. Wu, S. Huang, and A. Saied. Economic hyperparameter optimization with blended search strategy. In Proceedings of the International Conference on Learning Representations (ICLR’21), 2021. Published online: iclr.cc. find
  174. C. Wang, S. Liu, and A. Awadallah. Cost-effective hyperparameter optimization for large language model generation inference. arXiv:2303.04673 [cs.CL], 2023a. — [arXiv:2303.04673](https://arxiv.org/abs/2303.04673) find
  175. Y. Wang, Y. Kordi, S. Mishra, A., N. Smith, D. Khashabi, and H. Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 60th Conference of the North American Chapter of the Association for Computational Linguistics (ACL’23), 2023b. find
  176. Z. Wang, J. Wohlwend, and T. Lei. Structured pruning of large language models. In B. Webber, T. Cohn, Y. He, and Y. Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6151–6162. Association for Computational Linguistics, 2020.
  177. H. Weerts, F. Pfisterer, M. Feurer, K. Eggensperger, E. Bergman, N. Awad, J. Vanschoren, M. Pechenizkiy, B. Bischl, and F. Hutter. Can fairness be automated? guidelines and opportunities for fairness-aware automl. arXiv:2303.08485 [cs.AI], 2023. — [arXiv:2303.08485](https://arxiv.org/abs/2303.08485) find
  178. J. Wei, M Bosma, V. Zhao, K. Guu, A. Wei Yu, B. Lester, Du N, A. Dai, and Q. Le. Finetuned language models are zero-shot learners. In Proceedings of the International Conference on Learning Representations (ICLR’22), 2022. Published online: iclr.cc.
  179. C. White, A. Zela, B. Ru, Y. Liu, and F. Hutter. How powerful are performance predictors in neural architecture search? In M. Ranzato, A. Beygelzimer, K. Nguyen, P. Liang, J. Vaughan, and Y. Dauphin (eds.), Proceedings of the 35th International Conference on Advances in Neural Information Processing… find
  180. C. White, M. Safari, R. Sukthanker, B. Ru, T. Elsken, A. Zela, D. Dey, and F. Hutter. Neural architecture search: Insights from 1000 papers. arXiv:2301.08727, 2023. — [arXiv:2301.08727](https://arxiv.org/abs/2301.08727) find
  181. M. Wistuba, N. Schilling, and L. Schmidt-Thieme. Hyperparameter search space pruning - A new component for sequential model-based hyperparameter optimization. In A. Appice, P. Rodrigues, V. Costa, J. Gama, A. Jorge, and C. Soares (eds.), Machine Learning and Knowledge Discovery in Databases (ECML/PK… find
  182. M. Wistuba, A. Rawat, and T. Pedapati. A survey on neural architecture search. arXiv:1905.01392 [cs.LG], 2019. — [arXiv:1905.01392](https://arxiv.org/abs/1905.01392) find
  183. C. Wu, X. Zhang, Y. Zhang, Y. Wang, and W. Xie. PMC-LLaMA: Further finetuning LLaMA on medical papers. arXiv:2304.14454 [cs.CL], 2023a.
  184. J. Wu, W. Gan, Z. Chen, S. Wan, and P. S. Yu. Multimodal large language models: A survey. arXiv:2311.13165 [cs.AI], 2023b. — [arXiv:2311.13165](https://arxiv.org/abs/2311.13165)
  185. Z. Wu, Y. Hu, W. Shi, N. Dziri, A. Suhr, P. Ammanabrolu, N. Smith, M. Ostendorf, and H. Hajishirzi. Fine-grained human feedback gives better rewards for language model training. arXiv:2306.01693 [cs.CL], 2023c.
  186. R. Xiong, Y. Yang, D. He, K. Zheng, S. Zheng, C. Xing, H. Zhang, Y. Lan, L. Wang, and T. Liu. On layer normalization in the transformer architecture. In H. Daume III and A. Singh (eds.), Proceedings of the 37th International Conference on Machine Learning (ICML’20), volume 98. Proceedings of Machine… find
  187. T. Xu, P. Zhang, Q. Huang, H. Zhang, Z. Gan, X. Huang, and X. He. AttnGAN: Fine-grained text to image generation with attentional generative adversarial networks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1316–1324. find
  188. Computer Vision Foundation and IEEE Computer Society, IEEE, 2018. find
  189. G. Yang, E. Hu, I. Babuschkin, S. Sidor, X. Liu, D. Farhi, N. Ryder, J. Pachocki, W. Chen, and J. Gao. Tuning large neural networks via zero-shot hyperparameter transfer. In M. Ranzato, A. Beygelzimer, K. Nguyen, P. Liang, J. Vaughan, and Y. Dauphin (eds.), Proceedings of the 35th International Conf… find
  190. X. Yang, J. Ye, and X. Wang. Factorizing knowledge in neural networks. arXiv:2207.03337 [cs.CV], 2022. — [arXiv:2207.03337](https://arxiv.org/abs/2207.03337) find
  191. S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen. A survey on multimodal large language models. arXiv:2306.13549 [cs.CV], 2023.
  192. C. Ying, A. Klein, E. Christiansen, E. Real, K. Murphy, and F. Hutter. NAS-Bench-101: Towards reproducible Neural Architecture Search. In K. Chaudhuri and R. Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning (ICML’19), volume 97, pp. 7105–7114. Proceedings of… find
  193. C. Yu, X. Liu, C. Tang, W. Feng, and J. Lv. Gpt-nas: Neural architecture search with the generative pre-trained model. arXiv:2305.05351 [cs.CV], 2023a. — [arXiv:2305.05351](https://arxiv.org/abs/2305.05351) find
  194. L. Yu, D. Simig, C. Flaherty, A. Aghajanyan, L. Zettlemoyer, and M. Lewis. MEGABYTE: predicting million-byte sequences with multiscale transformers. arXiv:2305.07185 [cs.LG], 2023b. find
  195. C. Yuan and S. Agaian. A comprehensive review of binary neural network. Artificial Intelligence Review, 56 (11):12949–13013, 2023. find
  196. A. Zela, J. Siems, L. Zimmer, J. Lukasik, M. Keuper, and F. Hutter. Surrogate NAS benchmarks: Going beyond the limited search spaces of tabular NAS benchmarks. In Proceedings of the International Conference on Learning Representations (ICLR’22), 2022. Published online: iclr.cc. find
  197. L. Zhang, Y. Zhang, K. Ren, R. Li, and Y. Yang. MLCopilot: Unleashing the power of large language models in solving machine learning tasks. arXiv:2304.12210 [cs.LG], 2023a.
  198. R. Zhang, J. Han, C. Liu, P. Gao, A. Zhou, X. Hu, S. Yan, P. Lu, H. Li, and Y. Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv:2303.16199 [cs.CV], 2023b.
  199. S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. OPT: Open pre-trained transformer language models. arXiv:2205.01068 [cs.CL], 2022.
  200. S. Zhang, L. Dong, X. Li, S. Zhang, X. Sun, S. Wang, J. Li, R. Hu, T. Zhang, F. Wu, and G. Wang. Instruction tuning for large language models: A survey. arXiv:2308.10792 [cs.CL], 2023c.
  201. S. Zhang, C. Gong, L. Wu, X. Liu, and M. Zhou. AutoML-GPT: Automatic machine learning with GPT. arXiv:2305.02499 [cs.CL], 2023d.
  202. W. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, Y. Du, C. Yang, Y. Chen, Z. Chen, J. Jiang, R. Ren, Y. Li, X. Tang, Z. Liu, P. Liu, J. Nie, and J. Wen. A survey of large language models. arXiv:2303.18223 [cs.CL], 2023a.
  203. Y. Zhao, T. Pang, C. Du, X. Yang, C. Li, N. Cheung, and M. Lin. On evaluating adversarial robustness of large vision-language models. In Proceedings of the 37th International Conference on Advances in Neural Information Processing Systems (NeurIPS’23). Curran Associates, 2023b. find
  204. M. Zheng, X. Su, S. You, F. Wang, C. Qian, C. Xu, and S. Albanie. Can gpt-4 perform neural architecture search? arXiv:2304.10970 [cs.LG], 2023.
  205. Y. Zhou, A. Ioan Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, and J. Ba. Large language models are human-level prompt engineers. In International Conference on Learning Representations (ICLR’23), 2023. Published online: iclr.cc.
  206. L. Zimmer, M. Lindauer, and F. Hutter. Auto-Pytorch: Multi-fidelity metalearning for efficient and robust AutoDL. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43:3079–3090, 2021. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tornede-2023-automl-age-large-language-models/