AI Insight
This paper establishes a mathematically rigorous convergence result showing that randomly initialized wide neural networks approach Gaussian processes as their width increases. The authors prove that this convergence occurs at a rate of m^(-1/2) (where m is network width) using the supremum norm on compact input sets with polynomial activations, while maintaining explicit constants that remain bounded regardless of sample size. This addresses a key limitation of previous finite-sample bounds, whose constants degraded as more sample points were added, by measuring error per sample and then extending to functional convergence through dense refinement.
Why it matters
This theoretical advance provides stronger guarantees for understanding neural network behavior in the infinite-width limit, which is foundational for analyzing neural network training dynamics and generalization. The explicit, uniform constants make these bounds practically informative for understanding how wide a network must be to approximate Gaussian process behavior on realistic input domains.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Randomly initialised neural networks of width $m$ converge to Gaussian processes as $m to infty$, with two types of quantitative bounds characterising this convergence. Finite-sample bounds compare the network with its Gaussian limit at a fixed finite sample set, on an arbitrary domain, in the quadratic Wasserstein distance $mathcal W_2$, achieving a convergence rate of $m^{-1/2}$ with explicit constants. However, these constants grow with the number of samples, so the bound becomes uninformative as the sample set is refined. Functional bounds remove that dependence by comparing the two as random functions, at the price of spherical input, metrics weaker than $mathcal W_2$, or slower rates. In this paper, we therefore lift a finite-sample bound to a functional one. The lifting refines the sample set until it is dense in the domain, so the bound must first be made independent of the number of samples. We achieve this by measuring the error per sample, which makes the constant uniform. For a polynomial activation and any compact input set, this gives a rate of $m^{-1/2}$ in the supremum norm, with an explicit constant.
Source: Lifting Finite-Sample Gaussian Approximation of Wide Neural Networks to Functional Convergence