AI Insight
This study develops a framework to compare model-agnostic feature selection methods based on relative efficiency, focusing on two state-of-the-art approaches: Generalized Covariance Measure (GCM) and Leave-One-Covariate-Out (LOCO) estimation. The researchers provide theoretical analysis across linear models, non-linear additive models, and single index models, complemented by simulations and real data applications. Their results demonstrate that GCM-related methods generally outperform LOCO under specific regularity conditions, with the asymptotic relative efficiency quantified through a defined correlation quantity.
Why it matters
This research provides rigorous guidance for selecting appropriate feature selection methods in machine learning applications, including neural networks and gradient boosting trees. The framework helps practitioners make informed decisions about which model-agnostic feature selection approach to use based on their specific modeling context, potentially improving model interpretability and performance across various domains.
Understand the Science
arXiv:2508.14268v2 Announce Type: replace-cross
Abstract: Feature selection and importance estimation in a model-agnostic setting is an ongoing challenge of significant interest. Wrapper methods are commonly used because they are typically model-agnostic. In this paper, we develop a general comparison framework for model-agnostic feature selection methods based on relative efficiency, using emph{relative variability} $sigma/mu$ to account for different statistics having different means. In particular we focus on state-of-the-art feature selection methods, the Generalized Covariance Measure (GCM) and Leave-One-Covariate-Out (LOCO) estimation. In particular, we present a theoretical comparison under three model settings: linear models, non-linear additive models, and single index models that mimic a single-layer neural network. We complement this with simulations and real data examples for the above models and mis-specified models. Our theoretical results, along with empirical findings, demonstrate that GCM-related methods generally out-perform LOCO under suitable regularity conditions defined by a suitably defined correlation quantity which quantifies the asymptotic relative efficiency of these approaches. Our simulations and real data analysis include widely used machine learning methods such as neural networks and gradient boosting trees.
Source: Comparing Model-agnostic Feature Selection Methods through Relative Efficiency