Hyperparameters are distinct from model parameters, which are learned from data during training. Instead, hyperparameters are set prior to training and include choices such as the learning rate, the number of hidden layers and neurons in neural networks, or the regularization strength in regression models. They critically influence the behavior of the learning algorithm and its ability to generalize from training data to unseen data. Finding the optimal set of hyperparameters is a significant aspect of model development and is often achieved through processes like grid search, random search, or automated optimization techniques like Bayesian optimization.
The concept of hyperparameters has been integral to machine learning since its inception, but the term and its systematic tuning became more prominent with the rise of complex models such as deep neural networks in the 2000s. The need to efficiently search hyperparameter spaces has led to the development of more sophisticated optimization strategies beyond manual tuning.
No single individual is credited with the introduction of hyperparameters, as the concept evolved with the field of machine learning. However, researchers in the development of optimization techniques for hyperparameter tuning, such as those involved in the creation of Bayesian optimization methods, have been key in advancing how we select and evaluate these critical model settings.