🧑‍🔬 Timothy Lillicrap

Timothy Lillicrap works on neural network architectures for machine learning. The two papers here take up two related problems. One is how a network can answer questions that depend on relations between entities rather than on the entities alone. The other is how a sequence model can keep useful information over long spans without the cost of memory and attention growing out of hand. Both propose a component that can be added to existing models.

The more cited of the two is A simple neural network module for relational reasoning (2017), which introduced the Relation Network. A single shared function is applied to every pair of input objects, the pairwise outputs are summed, and a second network processes the total. This makes the module insensitive to object order and lets it learn one relation function for all pairs. The objects can be cells of an image feature map, encoded sentences, or rows of a state description.

The later paper, Compressive Transformers for Long-Range Sequence Modelling (2019), moves toward language modelling over long histories. Instead of discarding old activations once a fixed-size buffer fills, the model folds them into a second, coarser compressed memory, for instance by merging several slots with a strided convolution. Attention then reads from both memories. The compression step has a training objective of its own, which measures how closely attention on the condensed slots matches attention on the full, uncompressed ones.

Lillicrap is based at Google DeepMind in London, with earlier bylines at Google in Mountain View, University College London and the University of Oxford, and education at Queen's University in Kingston, Canada, until 2014. Co-authors on the papers here include Adam Santoro, Peter Battaglia, David Raposo, Mateusz Malinowski and Anna Potapenko. The work is listed under Reasoning.

Timothy Lillicrap has 2 papers in our library, in 2017–2019, most of them on Reasoning; often with Adam Santoro, Anna Potapenko, David Raposo.