🧑‍🔬 Thomas Unterthiner

Thomas Unterthiner works in deep learning, on the basic building blocks of neural networks and on how architectures developed for one kind of data can be carried over to another. Their papers here ask how design choices inside a network affect training speed and generalization, and whether image recognition really depends on convolutional structure. Both questions are tested through experiments on standard image classification benchmarks.

The most cited of these works is Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs), from 2015. It introduced an activation function that passes positive inputs through unchanged and, for negative inputs, bends smoothly toward a fixed negative floor. Pulling average activations closer to zero gives some of the effect of batch normalization at lower cost, and the paper backs this with an analysis based on the unit natural gradient. The second is An Image is Worth 16x16 Words, from 2020. It fed images, cut into small square patches and embedded as a sequence, into a Transformer encoder borrowed almost unchanged from language processing, and called the result the Vision Transformer (ViT).

The later work shifts attention from the behaviour of individual units to whole architectures and to training at scale. The ViT study reports that, when trained on ImageNet alone, the model falls slightly behind a ResNet of similar size. It attributes the gap to the missing assumptions of locality and translation that convolutions build in, and it examines what changes as training grows larger.

Earlier bylines place Unterthiner at Johannes Kepler University of Linz from 2013 to 2019, then at Google from 2020 to 2022. The most recent ones, from 2023 and 2024, are at Google DeepMind. Co-authors include Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Mostafa Dehghani and Jakob Uszkoreit.

Thomas Unterthiner has 2 papers in our library, in 2015–2020; often with Alexander Kolesnikov, Jakob Uszkoreit, Lucas Beyer.