Feature Importance
Techniques used to identify and rank the significance of input variables (features) in contributing to the predictive power of a ML model.
Feature importance is a critical concept in machine learning that involves evaluating and identifying which features (input variables) in a dataset contribute most significantly to the predictive accuracy of a model. Understanding feature importance helps in model interpretation, improving model efficiency by eliminating irrelevant features, and in feature engineering to enhance model performance. Different methods are used to assess feature importance, including model-specific approaches like the coefficients in linear models or the feature importance scores in tree-based models such as Random Forests and Gradient Boosting Machines, and model-agnostic methods like Permutation Feature Importance. The choice of method depends on the type of model used and the specific characteristics of the data.
The concept of feature importance has evolved alongside the development of machine learning models, becoming more prominent as models have increased in complexity. While the idea has been implicit in statistical models for decades, explicit methods for evaluating feature importance in complex models like decision trees began to gain popularity in the 1990s and 2000s.
While it's challenging to pinpoint specific individuals responsible for the concept of feature importance due to its broad and fundamental nature in machine learning, notable contributions have been made by researchers developing tree-based algorithms, such as Leo Breiman with Random Forests, and Friedman with Gradient Boosting Machines, where feature importance is an integral part of model interpretation.