Choosing Multiple Parameters for Support Vector Machines

OLIVIER CHAPELLEVLADIMIR VAPNIKOLIVIER BOUSQUETSAYAN MUKHERJEE

article2002Machine-mediated learning2,407 citations

Proposes a gradient descent framework to automatically tune large numbers of support vector machine hyperparameters and kernel scaling factors by minimizing generalization error bounds, enabling simultaneous kernel optimization and feature selection.

Listen

Support vector machines (SVMs) are widely used for classification and pattern recognition, yet their predictive accuracy depends heavily on selecting optimal model parameters. While standard grid-search techniques work well for tuning one or two settings, they become computationally intractable when dealing with multiple parameters, such as individual feature weights. The article sets out to develop and evaluate an automated gradient descent methodology that efficiently tunes large sets of parameters by directly minimizing theoretical estimates of SVM generalization error.

The researchers formulated an iterative optimization framework that alternates between standard SVM training and adjusting model parameters using the gradient of generalization error bounds, particularly the radius-margin bound and regularized span bounds. They smoothed these theoretical bounds to make them mathematically differentiable and tested the approach across diverse benchmarks. The evaluation included standard benchmark datasets, handwritten digit classification (USPS), high-dimensional cancer gene-expression data, and real-time computer vision tasks like face detection.

The findings show that the proposed method scales successfully to more than one hundred parameters without suffering from overfitting. In standard two-parameter benchmarks, gradient descent matched the accuracy of traditional grid-search cross-validation while requiring roughly 100 times fewer training iterations. Furthermore, using this method to optimize individual input feature scaling factors enabled effective feature selection. In leukemia microarray classification, reducing the feature space from over 7,000 genes to 30 genes achieved zero test classification errors, while in medulloblastoma outcome prediction, selecting 60 genes reduced error counts from 24 down to 15. In face detection, reducing inputs to just 60 principal components maintained high accuracy while eliminating prohibitive computational overhead.

These results demonstrate that model development timelines and computational costs can be dramatically decreased while simultaneously improving model explainability and operational efficiency. By automatically shrinking irrelevant feature weights toward zero, the approach naturally identifies critical variables in high-dimensional domains like genomics and accelerates real-time computer vision systems. Crucially, the analysis reveals that an error bound does not need to predict exact error rates with high numeric accuracy; it only needs to preserve the proper location of the optimal parameter values.

Organizations deploying SVMs should adopt gradient-based minimization of the radius-margin bound as a practical, computationally efficient alternative to exhaustive grid searches and span bounds, which present more local minima. Decision-makers should leverage this framework for automated feature selection before deploying complex models into production environments. However, leaders should note that the mathematical derivations assume smooth kernel differentiability and may encounter local minima. Further research is recommended to establish comprehensive theoretical error bounds for the overall automated procedure across broader non-linear problem domains.

AT&T Labs—Research École Polytechnique LIP6 Massachusetts Institute of Technology pattern recognition support vector svm Parameters Support Vector Machines Error

CHAPELLE et al (2002).pdf

Paper: A training algorithm for optimal margin classifiers, B. Boser et al. (1992). Reading this foundational work on optimal margin classifiers provides essential context on the dual-space optimization techniques that support vector machines rely on.
Paper: Support-vector networks, Corinna Cortes et al. (1995). Understanding this classic introduction to support-vector networks is necessary before exploring automated hyperparameter tuning for them.

Paper: Algorithms for Hyper-Parameter Optimization, James Bergstra et al. (2011). This book chapter extends the source work's focus on hyperparameter tuning by evaluating advanced sequential model-based optimization algorithms.
Paper: Practical Bayesian Optimization of Machine Learning Algorithms, Jasper Snoek et al. (2012). This chapter builds directly on the parameter tuning problem addressed in the source by developing practical Bayesian optimization procedures for machine learning algorithms.

Cover for Choosing Multiple Parameters for Support Vector Machines

Abstract

The problem of automatically tuning multiple parameters for pattern recognition Support Vector Machines (SVMs) is considered. This is done by minimizing some estimates of the generalization error of SVMs using a gradient descent algorithm over the set of parameters. Usual methods for choosing parameters, based on exhaustive search become intractable as soon as the number of parameters exceeds two. Some experimental results assess the feasibility of our approach for a large number of parameters (more than 100) and demonstrate an improvement of generalization performance.

Topics

pattern recognition support vector svm adaptive boosting control parameters

Keywords

Parameters Support Vector Machines Error Choosing Multiple Parameters Kernel Set SVM Training Feature Span

1. Introduction
2. Support vector learning
3. Estimating the performance of an SVM
3.1. Single validation estimate
3.2. Leave-one-out bounds
4. Smoothing the test error estimates
5. Optimizing the kernel parameters
6. Computing the gradient
6.1. Computing the derivative of the margin
6.2. Computing the derivative of the radius
6.3. Computing the derivative of the hyperplane parameters
6.4. Computing the derivative of the span-rule
7. Experiments
7.1. Optimization details
7.2. Benchmark databases
7.3. Automatic selection of scaling factors
8. Feature selection
8.1. Toy data
8.2. DNA microarray data
8.3. Face detection
9. Conclusion
Acknowledgments
Notes
References

Knowls

Knowl 1 — Alternating Optimization Framework for Support Vector Machine Hyperparameters
algorithm
The hyperparameter optimization framework tunes continuous kernel and regularization parameters $\boldsymbol{\theta} \in \mathbb{R}^n$ of a Support Vector Machine (SVM) by alternating standard quadratic programming (QP) training for the dual variables $\boldsymbol{\alpha}$ with gradient descent updates on $\boldsymbol{\theta}$ to minimize a differentiable generalization error bound or estimate $T(\boldsymbol{\alpha}^0(\boldsymbol{\theta}), \boldsymbol{\theta})$ . To prevent positivity constraints during optimization (for example, on bandwidth $\sigma$ or error penalty $C$ ), logarithmic reparameterizations such as $\boldsymbol{\theta} = (\log C, \log \sigma)$ are employed.
Input: Training dataset $Z = \{(\mathbf{x}_i, y_i)\}_{i=1}^\ell$ , initial hyperparameter vector $\boldsymbol{\theta}_0$ , learning rate $\varepsilon > 0$ , criterion function $T$
Output: Optimized hyperparameters $\boldsymbol{\theta}^*$ , optimal SVM dual coefficients $\boldsymbol{\alpha}^0$
Initialize $\boldsymbol{\theta} \leftarrow \boldsymbol{\theta}_0$
repeat
    Compute optimal dual coefficients $\boldsymbol{\alpha}^0(\boldsymbol{\theta})$ by solving the standard SVM quadratic program:
     $\boldsymbol{\alpha}^0(\boldsymbol{\theta}) = \arg\max_{\boldsymbol{\alpha}} \left( \sum_{i=1}^\ell \alpha_i - \frac{1}{2} \sum_{i,j=1}^\ell \alpha_i \alpha_j y_i y_j K_{\boldsymbol{\theta}}(\mathbf{x}_i, \mathbf{x}_j) \right)$
    subject to $\sum_{i=1}^\ell \alpha_i y_i = 0$ and $\alpha_i \ge 0$

    Evaluate the gradient $\nabla_{\boldsymbol{\theta}} T(\boldsymbol{\alpha}^0, \boldsymbol{\theta})$

    Update parameters via gradient step:
     $\boldsymbol{\theta} \leftarrow \boldsymbol{\theta} - \varepsilon \nabla_{\boldsymbol{\theta}} T(\boldsymbol{\alpha}^0, \boldsymbol{\theta})$
    (or Newton step $\boldsymbol{\theta} \leftarrow \boldsymbol{\theta} - (\Delta_{\boldsymbol{\theta}} T)^{-1} \nabla_{\boldsymbol{\theta}} T(\boldsymbol{\alpha}^0, \boldsymbol{\theta})$ where $(\Delta_{\boldsymbol{\theta}} T)_{ij} = \frac{\partial^2 T}{\partial \theta_i \partial \theta_j}$ )
until stopping criterion on $T$ is met
return $\boldsymbol{\theta}, \boldsymbol{\alpha}^0(\boldsymbol{\theta})$
Knowl 2 — Derivative of Parametric Constrained Quadratic Maxima
theoretical result
Let $\mathbf{v}_\theta \in \mathbb{R}^n$ and $\mathbf{P}_\theta \in \mathbb{R}^{n \times n}$ be a vector and matrix depending smoothly on a scalar parameter $\theta$ . Consider the constrained quadratic optimization objective:

$L(\theta) = \max_{\mathbf{x} \in \mathcal{F}} \left( \mathbf{x}^T \mathbf{v}_\theta - \frac{1}{2} \mathbf{x}^T \mathbf{P}_\theta \mathbf{x} \right)$

over the convex feasible domain $\mathcal{F} = \{\mathbf{x} \in \mathbb{R}^n : \mathbf{b}^T \mathbf{x} = c, \mathbf{x} \ge 0\}$ . If the maximizer $\bar{\mathbf{x}} = \arg\max_{\mathbf{x} \in \mathcal{F}} (\mathbf{x}^T \mathbf{v}_\theta - \frac{1}{2} \mathbf{x}^T \mathbf{P}_\theta \mathbf{x})$ is unique, the function $L(\theta)$ is differentiable with respect to $\theta$ , and its derivative is given by:

$\frac{\partial L(\theta)}{\partial \theta} = \bar{\mathbf{x}}^T \frac{\partial \mathbf{v}_\theta}{\partial \theta} - \frac{1}{2} \bar{\mathbf{x}}^T \frac{\partial \mathbf{P}_\theta}{\partial \theta} \bar{\mathbf{x}}$

This implies that the objective can be differentiated with respect to $\theta$ while treating the optimal vector $\bar{\mathbf{x}}$ as a constant. The result holds unchanged if either or both the equality constraint $\mathbf{b}^T \mathbf{x} = c$ and the non-negativity constraint $\mathbf{x} \ge 0$ are omitted.
Knowl 3 — Gradients of the Radius-Margin Bound with Respect to Kernel Parameters
theoretical result
For a hard-margin SVM (or a soft-margin SVM with quadratic error penalization converted to hard-margin via the modified kernel $\mathbf{K} \leftarrow \mathbf{K} + \frac{1}{C}\mathbf{I}$ ), the radius-margin generalization error bound is proportional to $R^2 \|\mathbf{w}\|^2 = R^2 / \gamma^2$ . Here, $R$ is the radius of the minimal enclosing sphere in feature space, and $\gamma = 1/\|\mathbf{w}\|$ is the margin.

The partial derivative of the squared norm of the hyperplane weights $\|\mathbf{w}\|^2$ with respect to a kernel parameter $\theta_p$ is:

$\frac{\partial \|\mathbf{w}\|^2}{\partial \theta_p} = -\sum_{i,j=1}^\ell \alpha_i^0 \alpha_j^0 y_i y_j \frac{\partial K(\mathbf{x}_i, \mathbf{x}_j)}{\partial \theta_p}$

where $\boldsymbol{\alpha}^0$ is the vector of optimal dual variables maximizing $W(\boldsymbol{\alpha})$ .

The squared radius $R^2$ is computed by maximizing $\sum_{i=1}^\ell \beta_i K(\mathbf{x}_i, \mathbf{x}_i) - \sum_{i,j=1}^\ell \beta_i \beta_j K(\mathbf{x}_i, \mathbf{x}_j)$ subject to $\sum_{i=1}^\ell \beta_i = 1$ and $\beta_i \ge 0$ . Its partial derivative with respect to $\theta_p$ is:

$\frac{\partial R^2}{\partial \theta_p} = \sum_{i=1}^\ell \beta_i^0 \frac{\partial K(\mathbf{x}_i, \mathbf{x}_i)}{\partial \theta_p} - \sum_{i,j=1}^\ell \beta_i^0 \beta_j^0 \frac{\partial K(\mathbf{x}_i, \mathbf{x}_j)}{\partial \theta_p}$

where $\boldsymbol{\beta}^0$ is the optimal solution to the minimal enclosing sphere quadratic program.
Knowl 4 — Regularized Span of Support Vectors and Its Closed-Form Derivative
model/method
The unregularized span of support vectors $S_p^2 = 1/(\tilde{\mathbf{K}}_{SV}^{-1})_{pp}$ exhibits jump discontinuities whenever the set of support vectors changes during hyperparameter updates. To make the span continuous and differentiable for gradient descent, a regularization penalty is introduced into the span definition for each support vector $\mathbf{x}_p$ :

$S_p^2 = \min_{\boldsymbol{\lambda},\, \sum_{i \ne p} \lambda_i = 1} \left\| \Phi(\mathbf{x}_p) - \sum_{i \ne p} \lambda_i \Phi(\mathbf{x}_i) \right\|^2 + \eta \sum_{i \ne p} \frac{1}{\alpha_i^0} \lambda_i^2$

where $\eta > 0$ is a regularization parameter (typically set to $\eta = 0.1$ ), $\Phi(\mathbf{x})$ is the feature map, and $\alpha_i^0$ are the SVM dual variables.

In matrix notation, the regularized span for support vector $p$ is computed in closed form as:

$S_p^2 = \frac{1}{(\tilde{\mathbf{K}}_{SV} + \mathbf{D})^{-1}_{pp}} - D_{pp}$

where $\tilde{\mathbf{K}}_{SV} = \begin{pmatrix} \mathbf{K}_{SV} & \mathbf{1} \\ \mathbf{1}^T & 0 \end{pmatrix}$ , $\mathbf{K}_{SV}$ is the kernel matrix on the support vectors, and $\mathbf{D}$ is a diagonal matrix with diagonal entries $D_{ii} = \eta / \alpha_i^0$ for $i \le n_{SV}$ and $D_{n_{SV}+1, n_{SV}+1} = 0$ .

The derivative of the regularized span with respect to a kernel parameter $\theta_p$ is:

$\frac{\partial S_p^2}{\partial \theta_p} = S_p^4 \left[ (\tilde{\mathbf{K}}_{SV} + \mathbf{D})^{-1} \frac{\partial \tilde{\mathbf{K}}_{SV}}{\partial \theta_p} (\tilde{\mathbf{K}}_{SV} + \mathbf{D})^{-1} \right]_{pp}$

When $\eta \to 0$ , this estimate recovers the Opper-Winther bound, and when $\eta \to \infty$ , it recovers the Jaakkola-Haussler bound.
Knowl 5 — Exact Gradient of Hyperplane Parameters with Respect to Kernel Parameters
equation
When non-support vectors are omitted, the active Karush-Kuhn-Tucker (KKT) equality conditions on the margin for an SVM with threshold $b$ and $n$ support vectors can be written as the linear system:

$\mathbf{H} \begin{pmatrix} \boldsymbol{\alpha}^0 \\ b \end{pmatrix} = \begin{pmatrix} \mathbf{1} \\ 0 \end{pmatrix}, \quad \text{where } \mathbf{H} = \begin{pmatrix} \mathbf{K}_Y & \mathbf{y} \\ \mathbf{y}^T & 0 \end{pmatrix}$

Here, $(\mathbf{K}_Y)_{ij} = y_i y_j K(\mathbf{x}_i, \mathbf{x}_j)$ , $\mathbf{y} = (y_1, \dots, y_n)^T$ , and $\mathbf{1} = (1, \dots, 1)^T \in \mathbb{R}^n$ .

Differentiating the identity $(\boldsymbol{\alpha}^0, b)^T = \mathbf{H}^{-1} (\mathbf{1}^T, 0)^T$ using the matrix inverse derivative $\frac{\partial \mathbf{M}^{-1}}{\partial \theta_p} = -\mathbf{M}^{-1} \frac{\partial \mathbf{M}}{\partial \theta_p} \mathbf{M}^{-1}$ yields:

$\frac{\partial (\boldsymbol{\alpha}^0, b)^T}{\partial \theta_p} = -\mathbf{H}^{-1} \frac{\partial \mathbf{H}}{\partial \theta_p} \begin{pmatrix} \boldsymbol{\alpha}^0 \\ b \end{pmatrix}$
Knowl 6 — Gradient-Based Feature Selection via Kernel Scaling Factors
algorithm
Feature selection is conducted by assigning individual scaling factors $\theta_i$ to each input feature (or principal component) within the kernel function: $K_{\boldsymbol{\theta}}(\mathbf{x}, \mathbf{z}) = K(\boldsymbol{\theta}^T \mathbf{x}, \boldsymbol{\theta}^T \mathbf{z})$ in input space, or $K_{\boldsymbol{\theta}}(\mathbf{x}, \mathbf{z}) = K(\boldsymbol{\theta}^T \boldsymbol{\Sigma} \mathbf{x}, \boldsymbol{\theta}^T \boldsymbol{\Sigma} \mathbf{z})$ in principal component space where $\boldsymbol{\Sigma}$ is the principal component projection matrix. Features associated with scaling factors that become sufficiently small during gradient descent are iteratively eliminated.
Input: Training dataset $Z = \{(\mathbf{x}_i, y_i)\}_{i=1}^\ell$ , input dimensionality $n$ , target number of features $m$ or pruning threshold, error bound criterion $T$
Output: Reduced feature set and trained SVM model
Initialize scaling vector $\boldsymbol{\theta} \leftarrow (1, 1, \dots, 1) \in \mathbb{R}^n$
If scaling in principal component space, compute PCA projection matrix $\boldsymbol{\Sigma}$
repeat
    Solve the standard SVM quadratic optimization problem using kernel $K_{\boldsymbol{\theta}}$
    repeat
        Update $\boldsymbol{\theta}$ using a gradient descent step on $T(\boldsymbol{\alpha}^0, \boldsymbol{\theta})$ with respect to $\boldsymbol{\theta}$
        Re-solve the SVM quadratic optimization problem with updated $\boldsymbol{\theta}$
    until a local minimum of $T$ is reached
    Discard feature dimensions corresponding to the smallest scaling coefficients $\theta_i$
    Re-initialize or adjust $\boldsymbol{\theta}$ on the remaining feature subspace
until desired number of features or convergence criteria is met
return selected features and final SVM classifier
Knowl 7 — Smoothed Validation Error via SVM Posterior Class Probabilities
model/method
To compute gradients of validation error when using a held-out validation set $\{(\mathbf{x}_i', y_i')\}_{i=1}^{n_v}$ , the non-differentiable step 0-1 loss is smoothed by estimating posterior class probabilities using Platt's sigmoid model on the continuous SVM output $f(\mathbf{x})$ :

$\tilde{P}_{A,B}(Y=1 \mid X=\mathbf{x}) = \frac{1}{1 + \exp(A f(\mathbf{x}) + B)}$

The parameters $A^*$ and $B^*$ are fitted via second-order gradient descent by minimizing the Kullback-Leibler divergence on the validation set:

$(A^*, B^*) = \arg\max_{A,B} \sum_{i=1}^{n_v} \left[ \frac{1 + y_i'}{2} \log \tilde{P}_{A,B}(\mathbf{x}_i') + \frac{1 - y_i'}{2} \log(1 - \tilde{P}_{A,B}(\mathbf{x}_i')) \right]$

The smoothed empirical generalization error $T$ to be minimized with respect to kernel parameters is then evaluated as:

$T = \sum_{i=1}^{n_v} \min\left( \tilde{P}_{A^*, B^*}(\mathbf{x}_i'),\, 1 - \tilde{P}_{A^*, B^*}(\mathbf{x}_i') \right)$

Knowl 8 — Generalization Error and Computational Cost of Bound Minimization vs Cross-Validation

data/table

Gradient descent optimization of the radius-margin bound ( $R^2/\gamma^2$ ) and the span bound was compared against standard 5-fold cross-validation on 5 benchmark datasets for selecting the soft-margin constant $C$ and RBF kernel width $\sigma$ . Across all datasets, bound minimization achieved test error rates matching 5-fold cross-validation while requiring approximately 100 times fewer SVM training iterations (3 to 14.2 training runs per split for $R^2/\gamma^2$ and 3.4 to 11.6 for the span bound, compared to 500 training runs for 5-fold CV over a $10 \times 10$ grid).

	Test Error (%)			Average SVM Trainings
Dataset	Cross-validation	$R^2/\gamma^2$	Span-bound	Cross-validation	$R^2/\gamma^2$	Span-bound
Breast cancer	$26.04 \pm 4.74$	$26.84 \pm 4.71$	$25.59 \pm 4.18$	500	14.2	7.0
Diabetes	$23.53 \pm 1.73$	$23.25 \pm 1.70$	$23.19 \pm 1.67$	500	12.2	9.8
Heart	$15.95 \pm 3.26$	$15.92 \pm 3.18$	$16.13 \pm 3.11$	500	9.0	6.2
Thyroid	$4.80 \pm 2.19$	$4.62 \pm 2.03$	$4.56 \pm 1.97$	500	3.0	11.6
Titanic	$22.42 \pm 1.02$	$22.88 \pm 1.23$	$22.50 \pm 0.88$	500	6.8	3.4

Knowl 9 — Feature Selection on Synthetic Linear and Non-Linear Benchmarks
empirical result
The gradient descent feature selection method based on $R^2/\gamma^2$ and regularized span bounds was evaluated on synthetic datasets containing irrelevant and redundant dimensions, keeping the top two selected features:
1. Linear problem (202 dimensions total, 6 relevant): Dimensions $x_1, x_2, x_3$ were drawn as $x_i = y \mathcal{N}(i, 1)$ and $x_4, x_5, x_6$ as $\mathcal{N}(0, 1)$ with probability 0.7 (and vice versa with probability 0.3); remaining features $x_7, \dots, x_{202} \sim \mathcal{N}(0, 20)$ . Using a linear kernel, the bound minimization selection method strongly outperformed standard SVMs and filter methods (Pearson correlation, Fisher score, Kolmogorov-Smirnov).
2. Non-linear problem (52 dimensions total, 2 relevant): $y = -1$ examples were drawn from equal-probability mixtures of $\mathcal{N}(\boldsymbol{\mu}_1, \mathbf{I})$ and $\mathcal{N}(\boldsymbol{\mu}_2, \mathbf{I})$ with $\boldsymbol{\mu}_1 = (-3/4, -3)$ and $\boldsymbol{\mu}_2 = (3/4, 3)$ ; $y = 1$ examples were drawn from $\boldsymbol{\mu}_1 = (3, -3)$ and $\boldsymbol{\mu}_2 = (-3, 3)$ . Noise features $x_3, \dots, x_{52} \sim \mathcal{N}(0, 20)$ . Using a second-order polynomial kernel, bound minimization successfully isolated the two true informative dimensions, while standard SVMs and filter methods (except Kolmogorov-Smirnov) suffered severe degradation.
Knowl 10 — Feature Selection on High-Dimensional DNA Microarray and Face Detection Applications
empirical result
Feature selection using gradient descent on the $R^2/\gamma^2$ and span bounds was evaluated on high-dimensional real-world tasks:
1. Leukemia Classification (Myeloid vs. Lymphoblastic, 7129 genes, 38 train / 34 test): Standard linear SVM achieved 1 test error. Gradient descent on $R^2/\gamma^2$ achieved 0 test errors using 30 selected genes (and 1 test error with just 1 gene), whereas Fisher criterion feature selection resulted in 1 test error for both 1 and 30 genes.
2. Lymphoblastic Cell Subtyping (B vs. T cells, 7129 genes): Standard linear SVM made 1 error. Both $R^2/\gamma^2$ and span bound minimization achieved 0 errors using 5 selected genes, compared to 2 errors for the Fisher criterion using 5 genes.
3. Medulloblastoma Treatment Outcome (7129 genes, 60 examples): Evaluated via leave-one-out error. Standard Gaussian SVM made 24 errors, whereas selecting 60 genes via $R^2/\gamma^2$ gradient descent reduced the error to 15.
4. Face Detection (CMU dataset, 479 faces, 57,000,000 non-face patterns): Selecting features via scaling factors in PCA space reduced the input dimension from 283 gray features to 60 principal components without loss of classification performance on the ROC curve, enabling real-time scanning.

Coverage note — None was omitted; all key theoretical derivations, bounds, algorithms, smoothing methods, and experimental datasets from the paper are represented.

References

1.Bengio. Y. (2000). Gradient-based optimization of hyper-parameters. Neural Computation, 12:8.
2.Bonnans J. F. & Shapiro, A. (2000). Perturbation analysis of optimization problems. Berlin: Springer-Verlag.
3.Chapelle, O. & Vapnik, V. (1999). Model selection for support vector machines. In Advances in neural information processing systems.
4.Cortes, C. & Vapnik, V. (1995). Support vector networks. Machine Learning, 20, 273–297.
5.Cristianini, N., Campbell, C., & Shawe-Taylor, J. (1999). Dynamically adapting kernels in support vector machines. In Advances in neural information processing systems.
6.Cristianini, N. & Shawe-Taylor, J. (2000).An introduction to support vector machines.Cambridge, MA: Cambridge University Press.
7.Golub, T., Slonim, D., Tamayo, P., Huard, C., Gaasenbeek, M., Mesirov, J. P., Coller, H., Loh, M. L., Downing, J. R., Caligiuri, M. A., Bloomfield, C. D., & Lander, E. S. (1999). Molecular classification of cancer: Class discovery and class prediction by gene expression monitoring. Science, 286, 531–537.
8.Heisele, B., Poggio, T., & Pontil, M. (2000). Face detection in still gray images. AI Memo 1687, Massachusetts Institute of Technology.
9.Jaakkola, T. S. & Haussler, D. (1999). Probabilistic kernel regression models. In Proceedings of the 1999 Conference on AI and Statistics.
10.Joachims, T. (2000). Estimating the generalization performance of a svm efficiently. In Proceedings of the International Conference on Machine Learning. San Mateo, CA: Morgan Kaufman.
11.Larsen, J., Svarer, C., Andersen, L. N., & Hansen, L. K. (1998). Adaptive regularization in neural network modeling. In G. B. Orr & K. R. Müller (Eds.). Neural networks: Trick of the trade. Berlin: Springer.
12.Luntz, A. & Brailovsky, V. (1969). On estimation of characters obtained in statistical procedure of recognition. Technicheskaya Kibernetica, 3, (in Russian).
13.Lütkepohl, H. (1996). Handbook of matrices. New York: Wiley & Sons.
14.Opper, M. & Winther, O. (2000). Gaussian processes and svm: Mean field and leave-one-out. In A. J. Smola, P. L. Bartlett, B. Schölkopf, & D. Schuurmans (Eds.). Advances in large margin classifiers (pp. 311–326). Cambridge, MA: MIT Press.
15.Platt, J. (2000). Probabilities for support vector machines. In A. Smola, P. Bartlett, B. Schölkopf, & D. Schuurmans (Eds.). Advances in large margin classifiers. Cambridge, MA: MIT Press.
16.Rätsch, G., Onoda, T., & Müller, K.-R. (2001). Soft margins for AdaBoost. Machine Learning, 42:3, 287–320.
17.Serre, T., Heisele, B., Mukherjee, S., & Poggio, T. (2000). Feature selection for face detection. AI Memo 1697, Massachusetts Institute of Technology.
18.Vapnik, V. (1995). The nature of statistical learning theory. Berlin: Springer.
19.Vapnik, V. (1998). Statistical learning theory. New York: John Wiley & Sons.
20.Vapnik, V. & Chapelle, O. (2000). Bounds on error expectation for support vector machines: Neural Computation, 12:9.
21.Wahba, G., Lin, Y., & Zhang, H. (2000). Generalized approximate crossvalidation for support vector machines: Another way to look at marginlike quantities. In A. Smola, P. Bartlett, B. Schölkopf, & D. Schuurmans (Eds.). Advances in large margin classifiers (pp. 297–309). Cambridge, MA: MIT Press.
22.Weston, J., Mukherjee, S., Chapelle, O., Pontil, M., Poggio, T., & Vapnik, V. (2000). Feature selection for support vector machines. In Advances in neural information processing systems.

Citation

MLA

Chapelle, O., et al. “Choosing Multiple Parameters for Support Vector Machines”. Machine Learning, vol. 46, nos. 1-3, 2002, pp. 131–59, https://doi.org/10.1023/A:1012450327387.

APA

Chapelle, O., Vapnik, V., Bousquet, O., & Mukherjee, S. (2002). Choosing Multiple Parameters for Support Vector Machines. Machine Learning, 46(1-3), 131–159. https://doi.org/10.1023/A:1012450327387

Chicago

Chapelle, O., V. Vapnik, O. Bousquet, and S. Mukherjee. 2002. “Choosing Multiple Parameters for Support Vector Machines”. Machine Learning 46 (1-3): 131–59. https://doi.org/10.1023/A:1012450327387.

Harvard

Chapelle, O. et al. (2002) “Choosing Multiple Parameters for Support Vector Machines”, Machine Learning, 46(1-3), pp. 131–159. Available at: https://doi.org/10.1023/A:1012450327387.

Vancouver

1. Chapelle O, Vapnik V, Bousquet O, Mukherjee S (2002) Choosing Multiple Parameters for Support Vector Machines. Machine Learning 46:131–159

BibTeX

@article{Chapelle_2002, title={Choosing Multiple Parameters for Support Vector Machines}, volume={46}, ISSN={1573-0565}, url={http://dx.doi.org/10.1023/A:1012450327387}, DOI={10.1023/a:1012450327387}, number={1-3}, journal={Machine Learning}, publisher={Springer Science and Business Media LLC}, author={Chapelle, Olivier and Vapnik, Vladimir and Bousquet, Olivier and Mukherjee, Sayan}, year={2002}, month=Jan, pages={131–159} }

Metadata:Crossref

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

Choosing Multiple Parameters for Support Vector Machines

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Alternating Optimization Framework for Support Vector Machine Hyperparameters

Knowl 2 — Derivative of Parametric Constrained Quadratic Maxima

Knowl 3 — Gradients of the Radius-Margin Bound with Respect to Kernel Parameters

Knowl 4 — Regularized Span of Support Vectors and Its Closed-Form Derivative

Knowl 5 — Exact Gradient of Hyperplane Parameters with Respect to Kernel Parameters

Knowl 6 — Gradient-Based Feature Selection via Kernel Scaling Factors

Knowl 7 — Smoothed Validation Error via SVM Posterior Class Probabilities

Knowl 8 — Generalization Error and Computational Cost of Bound Minimization vs Cross-Validation

Knowl 9 — Feature Selection on Synthetic Linear and Non-Linear Benchmarks

Knowl 10 — Feature Selection on High-Dimensional DNA Microarray and Face Detection Applications

References

Citation

BibTeX

Access the Paper