The Network Does Not Have to Wake Up Random
TL;DR for operators Training does not always have to begin with a model staring at the data like it has just been born in a dark room. The paper behind this article, S-GAI: Spectral Geometry-Aware Initialization for Sigmoidal MLPs, proposes a way to initialize a one-hidden-layer sigmoidal MLP from the geometry already visible in the training data.1 Instead of drawing hidden-layer weights randomly and asking gradient descent to discover everything later, S-GAI computes class-wise SVD structure, turns retained spectral directions into paired sigmoid “slab” gates, and uses those gates as the starting hidden representation. ...