Dimensionality Reduction via Sparse Support Vector Machines
Jinbo Bi, Kristin Bennett, Mark Embrechts, Curt Breneman, Minghu Song;
3(Mar):1229-1243, 2003.
Abstract
We describe a methodology for performing variable ranking and
selection using support vector machines (SVMs). The method
constructs a series of sparse linear SVMs to generate linear
models that can generalize well, and uses a subset of nonzero
weighted variables found by the linear models to produce a final
nonlinear model. The method exploits the fact that a linear SVM
(no kernels) with
l1-norm regularization inherently performs
variable selection as a side-effect of minimizing capacity of the
SVM model. The distribution of the linear model weights provides a
mechanism for ranking and interpreting the effects of variables.
Starplots are used to visualize the magnitude and variance of the
weights for each variable. We illustrate the effectiveness of
the methodology on synthetic data, benchmark problems, and
challenging regression problems in drug design. This method can
dramatically reduce the number of variables and outperforms SVMs
trained using all attributes and using the attributes selected
according to correlation coefficients. The visualization of the
resulting models is useful for understanding the role of
underlying variables.
[abs]
[pdf]
[ps.gz]
[ps]
[data]