Regularize What the Network Activates
TL;DR for operators A neural network can be regularized without acting only on its weights or randomly disabling units. Gaussian Neural Networks1 instead teaches hidden layers what their own normal activation patterns look like, then adds a penalty when internal signals become improbable under those learned distributions. That mechanism matters because the simpler implementation performed better than its complexity might suggest. Across ten regression and classification datasets, the sparse Gaussian neural network beat the corresponding basic ANN on nine and tied on one. The denser variance model, despite representing richer dependencies, was less consistent and could spend substantial training time optimizing the auxiliary objective rather than the task itself. ...