Analyzing Random Features via Linearization and the Matrix Dyson Equation

Abstract: Inspired by recent work using random matrix theory to study neural networks, we use the linearization trick and the matrix Dyson equation to find the limiting spectral distribution for the covariance matrix of Gaussian random features. Compared to prior work, this approach leads to a much shorter derivation of fixed point equations specifying the Stieltjes transform of the target distribution, which can then be solved numerically. Our empirical results suggest that our method achieves a faithful approximation to the limiting distribution that is closely matched by the empirical spectral distribution as dimension grows.