Overparameterization Regime
A phase in ML where the model has more parameters than the number of training samples, often leading to a high-variance, overfitted model.
The Overparameterization Regime refers to a situation in training machine learning (ML) models where the number of parameters in the model surpasses the number of training samples. This might seem counterintuitive, as many assume that it will inevitably lead to overfitting – memories of training data instead of learning to generalize – but recent theoretical and empirical studies suggest otherwise. In the Overparameterization Regime, models with excessive capacity can still generalize well even though they memorize noise and could potentially overfit. This concept has widespread implications in deep learning, an area that often deals with overparameterized models.
Overparameterization in machine learning models has been observed since the early days of artificial neural networks. However, understanding its implications and hence the term 'Overparameterization Regime,' saw a surge in interest with the advent and popularity of deep learning post-2010. While its exact origin is hard to pinpoint, its usage in literature gained momentum during the mid-2010s.
The term and the underlying principles are not attributed to a single individual or group. Instead, it is the collective output of the AI research community working on understanding the counter-intuitive generalization abilities of overparameterized models. This includes teams from universities, research institutions and tech companies contributing to developing and refining the concept.