📄 Efficient Architecture Search by Network Transformation

Summary

Automated neural architecture search with reinforcement learning or evolution has matched human-designed models. However, it usually needs hundreds of GPUs, because every candidate network is trained from zero. This paper presents EAS (Efficient Architecture Search), which avoids that waste. Instead of building networks from scratch, an RL meta-controller edits an existing trained network. Each step either widens a layer or inserts a new one, using function-preserving operations from Net2Net. The modified network starts out computing exactly what its parent did and inherits its weights, so it only needs fine-tuning before evaluation. A bidirectional LSTM encodes the current architecture. Separate actor networks choose the edits, and the controller is trained with REINFORCE using validation accuracy as the reward. On CIFAR-10 and SVHN, with only 5 GPUs, EAS found plain CNNs (no skip-connections or branching) that beat comparable designs. It also produced DenseNet variants that are more accurate and smaller than the originals. The main lesson is that reusing knowledge between candidates can make architecture search affordable for small teams.

Key points

• Plain CNN results: the best plain CNN found by EAS (20 layers, 23.4M parameters) reaches 4.23% test error on CIFAR-10 with standard augmentation (C10+). It outperforms ResNet and several of its variants and comes close to DenseNet.
• Gains over similar designs:
  • Compared with human-designed plain networks, error falls from 7.25% to 4.23% on C10+ and from 2.35% to 1.73% on SVHN.
  • Compared with MetaQNN, the relative error reduction is 38.9% on C10+ and 16.0% on SVHN.
• Compute cost: the search used 5 GPUs and trained a few hundred networks. NAS, by contrast, used 800 GPUs and tens of thousands of networks, and among auto-designed models only NAS with post-processing does better.
• DenseNet experiments:
  • The search starts from DenseNet-BC (L = 40, k = 40), with the growth rate picked from 40 to 64.
  • On C10 (no augmentation), EAS gets 4.66% error versus 5.19% for the best original DenseNet, with about 44% fewer parameters.
  • On C10+, it gets 3.44% versus 3.46%, with about 58% fewer parameters.
• RL versus random search: in a controlled test of 5 deepening and 4 widening steps, the RL controller moved toward good regions of the search space much more reliably than random search did.

Interesting ideas

• I1. Search by editing, not by generating. Each new candidate is a modified copy of a trained parent that already computes the same function, so evaluating it requires only fine-tuning. This warm-start principle could reduce cost in any iterative model-design loop, including NLP model scaling with limited hardware.
• I2. One encoder, several specialised actors. A shared bidirectional recurrent encoder turns a variable-length layer sequence into embeddings, and small separate heads each handle one type of edit. This keeps the controller simple and makes it easy to add new action types later.
• I3. One shared per-layer decision head. The widening actor applies the same sigmoid classifier to each layer's hidden state to decide whether to widen it. This makes the decision independent of depth and could be reused for other per-element choices, such as which layers to prune or freeze.
• I4. Evaluation quality depends on how candidates are trained. The authors suggest that MetaQNN's best model was shallow because deep networks trained quickly from scratch looked worse than they really were. Inherited weights give a fairer assessment of deep candidates, which is a useful warning for any search that relies on short training budgets.
• I5. Include a random-search baseline. Comparing the learned controller with random search over the same transformation actions shows whether the policy itself adds value. This is a cheap sanity check worth adding to any automated design study.