Sakana AI unveils pc-alm: a new training method for deep neural networks

Sakana AI researchers have introduced PC-ALM, a novel training method that enables the training of 1000-layer neural networks using a layer-local approach, bypassing the need for backpropagation. The method, an extension of predictive coding (PC), integrates Lagrangian multipliers to maintain credit assignment across layers, achieving results close to backpropagation on MNIST. PC-ALM, which is available as open-source code under an MIT license, has been tested on small image benchmarks and shows improved performance over traditional PC methods, particularly in deep, narrow networks. The research team reports that PC-ALM matches backpropagation across various widths, depths, and activation functions, with a test accuracy of 77.75% on Fashion-MNIST, compared to 68.13% for PC. The method also demonstrates effectiveness on ResNet-18 models on CIFAR-10 and Tiny ImageNet. The study highlights the potential of PC-ALM as a viable alternative to backpropagation, offering a more stable training process with damped oscillations rather than monotonic gradient flow.