Toward Optimal Feature Selection

article1996ICML1,913 citations

Develops an information-theoretic filter algorithm that efficiently eliminates both irrelevant and redundant features, providing a theoretically grounded solution for high-dimensional classification tasks without the computational burden of wrapper methods.

Listen

Modern classification tasks frequently involve datasets with hundreds or thousands of variables, many of which are irrelevant or redundant. High-dimensional data significantly slows model training—often exponentially—and increases the risk of overfitting, especially when available training samples are limited. Existing methods to select useful variables either scale poorly due to computationally prohibitive iterative model training or fail to eliminate redundant, correlated features.

The article develops and evaluates an information-theoretic framework and an efficient heuristic algorithm to eliminate uninformative variables prior to model training without sacrificing predictive accuracy.

The proposed method operates as a preprocessing filter using backward elimination. Rooted in information theory, it measures information loss via cross-entropy (Kullback-Leibler distance) and leverages the concept of a Markov blanket—a subset of features that renders a target feature conditionally independent of the class and all other variables. Because computing exact Markov blankets across all features is computationally intractable, the authors developed a heuristic algorithm that conditions on a small set of strongly correlated features to approximate information loss. The approach was evaluated on synthetic benchmarks (Corral, LED-24), standard benchmark datasets (Congressional Voting, DNA), and two high-dimensional text classification subsets from the Reuters corpus comprising over 1,600 features, comparing downstream classification performance against Naive Bayes and C4.5 decision tree models.

The evaluation yielded several key findings: First, the filter drastically compressed feature spaces while maintaining or improving predictive accuracy, notably reducing the DNA dataset from 180 features down to 30 or 80 features with equal or improved model accuracy. Second, the backward elimination approach successfully handled complex feature interactions where traditional forward selection failed, such as correctly removing misleading correlated features in the Corral dataset to boost C4.5 decision tree accuracy from 81.2% to 100%. Third, in high-dimensional text datasets, the algorithm removed approximately 1,000 features (reducing feature count by roughly 60%) in about 1.5 hours on standard hardware, whereas wrapper-based alternatives would require estimated thousands of hours. Finally, on the DNA benchmark, the algorithm performed feature reduction in 6 to 15 minutes, representing a two-order-of-magnitude runtime reduction compared to a 15-hour wrapper approach that achieved comparable accuracy.

These findings demonstrate that information-theoretic filtering enables organizations to deploy machine learning on high-dimensional domains, such as text classification and bioinformatics, at a fraction of the computational overhead and runtime cost of existing wrapper methods. Because the selection process is decoupled from specific downstream classifiers, the reduced feature subset can be generated once and shared across multiple modeling algorithms without retraining.

Organizations handling massive feature sets should adopt this filtering approach as a standard preprocessing step to prune noisy and redundant inputs before deploying complex or computationally heavy classification algorithms. When maximum accuracy is paramount, teams can use this filter to rapidly discard the vast majority of irrelevant features before applying specialized wrapper searches on the remaining compact subset. Practitioners should also evaluate whether simple, unconditioned feature selection suffices for domains with minimal feature interaction to further save preprocessing time.

The primary limitation of the algorithm is its heuristic approximation of Markov blankets, which uses a fixed number of conditioning variables. Increasing the conditioning set size can fragment small training samples and cause non-monotonic fluctuations in estimated feature relevance. Consequently, users should exercise caution and tune the conditioning parameter appropriately when working with limited training data.

Stanford University efficient algorithm genetic selection information theory optimal feature selection feature elimination subset selection

Koller et al (1996).pdf

Paper: Irrelevant Features and the Subset Selection Problem, George H. John et al. (1994). This paper establishes the foundational definitions of strong and weak feature relevance and subset selection frameworks that provide direct conceptual motivation for information-theoretic feature elimination.
Paper: Supervised and Unsupervised Discretization of Continuous Features, James Dougherty et al. (1995). It provides essential background on using entropy-based discretization for machine learning features, a common preprocessing requirement for information-theoretic feature selection methods.
Paper: Multi-Interval Discretization of Continuous-Valued Attributes for Classification Learning, Usama M. Fayyad et al. (1993). It introduces principled information-theoretic and minimum description length criteria for continuous attribute partitioning, laying algorithmic groundwork for information-based feature processing.
Paper: Induction of Decision Trees, J. R. Quinlan (1986). It introduces the classic information gain criterion for evaluating attribute relevance from data, which underpins information-theoretic subset selection heuristics.

Paper: Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy, Hanchuan Peng et al. (2003). This work formalizes and refines information-theoretic subset selection by introducing the minimal-redundancy-maximal-relevance (mRMR) framework to approximate optimal statistical dependency.
Paper: Feature Selection for High-Dimensional Data: A Fast Correlation-Based Filter Solution, Lei Yu et al. (2003). It advances information-theoretic subset filtering by formulating the Fast Correlation-Based Filter (FCBF) using symmetrical uncertainty to rapidly eliminate redundant and irrelevant features.
Paper: Correlation-based Feature Selection for Discrete and Numeric Class Machine Learning, Mark A. Hall (1999). It extends fast subset-evaluation concepts by developing a correlation-based filter heuristic that explicitly penalizes inter-feature redundancy.
Paper: An Introduction to Variable and Feature Selection, Isabelle M Guyon et al. (2003). It synthesizes subsequent progress across feature ranking and subset selection algorithms into a comprehensive taxonomy and practical methodology.
Paper: Feature Selection: Evaluation, Application, and Small Sample Performance, Anil K. Jain et al. (1997). It systematically compares and benchmarks sequential search algorithms for feature subset selection across complex synthetic and real-world distributions.
Paper: Toward integrating feature selection algorithms for classification and clustering, Huan Liu et al. (2005). It builds an integrative categorization framework that unifies subsequent filter, wrapper, and hybrid feature selection methods across tasks.
Paper: An Extensive Empirical Study of Feature Selection Metrics for Text Classification, George Forman (2003). It conducts a large-scale empirical evaluation of information gain and alternative feature selection criteria on high-dimensional text data.

Cover for Toward Optimal Feature Selection

Abstract

In this paper, we examine a method for feature subset selection based on Information Theory. Initially, a framework for defining the theoretically optimal, but computationally intractable, method for feature subset selection is presented. We show that our goal should be to eliminate a feature if it gives us little or no additional information beyond that subsumed by the remaining features. In particular, this will be the case for both irrelevant and redundant features. We then give an efficient algorithm for feature selection which computes an approximation to the optimal feature selection criterion. The conditions under which the approximate algorithm is successful are examined. Empirical results are given on a number of data sets, showing that the algorithm effectively handles datasets with a very large number of features.

Topics

efficient algorithm genetic selection information theory classifiers wrapper approach local binary patterns

Keywords

optimal feature selection feature elimination subset selection high-dimensional datasets information theory induction algorithms target concept computational cost

1. Introduction
2. Theoretical Framework
3. An Approximate Algorithm
4. Results
5. Conclusions

Knowls

Knowl 1 — Cross-Entropy Criterion for Optimal Feature Subset Selection
equation
Let $F = \{F_1, \dots, F_n\}$ be the complete set of features, $C \in \{c_1, \dots, c_\ell\}$ be the discrete target class variable, and $G \subseteq F$ be a candidate subset of features. For any instance assignment $f = (f_1, \dots, f_n)$ to $F$ , let $f_G$ denote the projection of $f$ onto the variables in $G$ . The information loss incurred by restricting the feature set from $F$ to $G$ is measured by the expected Kullback-Leibler divergence (cross-entropy) between the true conditional distribution $\Pr(C \mid F = f)$ and the projected conditional distribution $\Pr(C \mid G = f_G)$ :

$\Delta_G = \sum_{f} \Pr(F = f) \, D(\Pr(C \mid F = f) \,\|\, \Pr(C \mid G = f_G))$

where the cross-entropy $D(\mu \,\|\, \sigma)$ for two distributions $\mu$ and $\sigma$ over the class space $\{c_1, \dots, c_\ell\}$ is:

$D(\mu \,\|\, \sigma) = \sum_{k=1}^\ell \mu(c_k) \log \frac{\mu(c_k)}{\sigma(c_k)}$

The theoretically optimal feature selection objective is to find a minimal subset $G \subseteq F$ for which $\Delta_G$ is minimized (with $\Delta_F = 0$ ).
Knowl 2 — Markov Blanket Condition for Zero-Divergence Feature Elimination
theoretical result
Let $F$ be the full feature set, $C$ be the target class variable, and $G \subseteq F$ be a current active feature subset.

A feature $F_i \in G$ is conditionally independent of $C$ given $G' = G \setminus \{F_i\}$ if and only if $\Delta_{G'} = \Delta_G$ , where $\Delta_G$ is the expected cross-entropy $\sum_{f} \Pr(f) D(\Pr(C \mid F = f) \,\|\, \Pr(C \mid G = f_G))$ .

A subset of features $M \subseteq F \setminus \{F_i\}$ is defined as a Markov blanket for $F_i$ if $F_i$ is conditionally independent of all other features $(F \setminus M \setminus \{F_i\})$ given $M$ :

$\Pr(F \setminus M \setminus \{F_i\} \mid M, F_i) = \Pr(F \setminus M \setminus \{F_i\} \mid M)$

If some subset $M \subseteq G$ is a Markov blanket for $F_i$ , then the target class $C$ is conditionally independent of $F_i$ given $M$ (i.e., $\Pr(C \mid M, F_i) = \Pr(C \mid M)$ ), which implies that:

$\Delta_{G \setminus \{F_i\}} = \Delta_G$

Therefore, removing any feature $F_i$ that possesses a Markov blanket within the remaining features causes zero information loss with respect to predicting the class distribution.
Knowl 3 — Invariance of Markov Blanket Redundancy Under Successive Feature Removal
theoretical result
Let $G \subseteq F$ be the current set of features. Suppose a previously removed feature $F_i \notin G$ has a Markov blanket within $G$ . If another feature $F_j \in G$ is removed based on a Markov blanket $M_j \subseteq G$ , then $F_i$ is guaranteed to retain a valid Markov blanket within the reduced set $G \setminus \{F_j\}$ .

Specifically, if $M_i \subseteq G$ was a Markov blanket for $F_i$ before removing $F_j$ :
1. If $F_j \notin M_i$ , then $M_i \subseteq G \setminus \{F_j\}$ remains a valid Markov blanket for $F_i$ .
2. If $F_j \in M_i$ , letting $M_i' = M_i \setminus \{F_j\}$ , the union $M_i' \cup M_j \subseteq G \setminus \{F_j\}$ is a valid Markov blanket for $F_i$ .
This invariance guarantees that removing an unnecessary feature (whether irrelevant or redundant) never renders a previously eliminated feature necessary again, validating monotonic backward elimination.
Knowl 4 — Approximate Markov Blanket Feature Elimination Algorithm
algorithm
The algorithm approximates Markov blanket backward elimination by identifying a candidate conditioning set $M_i$ of $K$ features strongly correlated with $F_i$ and evaluating its expected cross-entropy.
Input: Full feature set $F = \{F_1, \dots, F_n\}$ , dataset of $m$ instances, class variable $C$ , conditioning set size $K \ge 0$ , number of features to eliminate $r$
Output: Reduced feature subset $G \subseteq F$ of size $n - r$
for each pair of features $F_i, F_j \in F$ with $i \ne j$ :
    Compute Pearson correlation:
     $\rho_{ij} = \frac{\text{Cov}(F_i, F_j)}{\text{Stddev}(F_i) \, \text{Stddev}(F_j)}$
Initialize active feature set $G = F$
for step = 1 to $r$ :
    for each feature $F_i \in G$ :
        Let $M_i \subseteq G \setminus \{F_i\}$ be the set of $K$ features $F_j \in G \setminus \{F_i\}$ with the largest correlation magnitude $|\rho_{ij}|$
        Compute expected cross-entropy:
         $\delta_G(F_i \mid M_i) = \sum_{f_{M_i}, f_i} \Pr(M_i = f_{M_i}, F_i = f_i) \, D(\Pr(C \mid M_i = f_{M_i}, F_i = f_i) \,\|\, \Pr(C \mid M_i = f_{M_i}))$
    Select feature $F^* = \arg\min_{F_i \in G} \delta_G(F_i \mid M_i)$
    Update active set: $G = G \setminus \{F^*\}$
return $G$
If $M_i$ is a true Markov blanket for $F_i$ , $\delta_G(F_i \mid M_i) = 0$ . The feature with the smallest value is greedily removed at each iteration.
Knowl 5 — Computational Complexity of Markov Blanket Filter Feature Selection
theoretical result
For a dataset with $n$ initial features, $m$ instances, $c$ classes, and an elimination target of $r$ features with a conditioning set size of $K$ features:
1. Correlation Matrix Computation: Computing all pairwise Pearson correlation factors $\rho_{ij}$ and sorting them for each feature requires: $O(n^2(m + \log n))$
2. Iterative Elimination: Evaluating the candidate Markov blanket $M_i$ and the expected cross-entropy over $2^K$ value configurations across all active features over $r$ elimination steps requires: $O(r \cdot n \cdot K \cdot m \cdot 2^K \cdot c)$
3. Caching Optimization: Because eliminating a feature $F^*$ only alters $M_i$ for the few remaining features that contained $F^*$ in their top- $K$ correlations, caching allows the elimination phase complexity to be reduced by approximately a factor of $n$ .
Unlike wrapper methods that require running induction algorithms $O(2^n)$ or $O(r \cdot n)$ times, this filter method is classifier-independent and scales efficiently to thousands of features.
Knowl 6 — Theoretical Justification of Backward Elimination over Forward Selection
model/method
Feature selection can be viewed as navigation in probability distribution space between the prior class distribution $\Pr(C)$ (no features) and the full conditional distribution $\Pr(C \mid F)$ (all features).

Backward elimination begins at $\Pr(C \mid F)$ , where cross-entropy $\Delta_F = 0$ , and removes features that cause the minimal increase in $\Delta_G$ . Because each elimination step explicitly bounds divergence from the full conditional distribution, the feature set stays close to the true target distribution throughout the search trajectory.

In contrast, forward selection begins at $\Pr(C)$ and greedily adds the feature $F_j$ maximizing information gain (expected cross-entropy between $\Pr(C \mid G)$ and $\Pr(C \mid G \cup \{F_j\})$ ). While this takes large steps away from $\Pr(C)$ , taking a large step away from the prior distribution does not guarantee moving closer to the target distribution $\Pr(C \mid F)$ , often leading forward selection into suboptimal subsets.
Knowl 7 — Limitations of Heuristic Markov Blanket Approximation
limitation
The heuristic Markov blanket feature selection algorithm exhibits two primary limitations as the conditioning set size $K$ varies:
1. Non-Monotonicity of Conditional Independence: Conditional independence is non-monotonic: a feature $F_i$ may be conditionally independent of class $C$ given a candidate blanket $M$ , yet strongly correlated with $C$ given a superset $M \cup \{F_j\}$ . As a result, selecting conditioning variables solely based on pairwise correlation can misjudge feature relevance, causing estimated relevance to fluctuate unpredictably as $K$ increases.
2. Data Sample Fragmentation: Evaluating expected cross-entropy requires estimating probabilities conditioned on joint assignments of $M_i$ . For discrete binary features, the sample space splits into $2^K$ bins. Increasing $K$ fragments finite training data into small partitions, degrading probability estimates and cross-entropy reliability.

Knowl 8 — Classification Performance on Benchmark UCI and Synthetic Datasets

data/table

The approximate Markov blanket feature selection algorithm was evaluated using Naive Bayes and C4.5 decision tree classifiers across synthetic and UCI benchmark datasets, comparing original feature sets against forward selection and backward elimination with conditioning set sizes $K \in \{0, 1, 2, 3, 4\}$ .

Dataset	# Features	$K$	Naive Bayes Accuracy			C4.5 Accuracy
	Orig. / Final		Orig.	Fwd.	Bckwd.	Orig.	Fwd.	Bckwd.
Corral	6 / 4	0	90.6%	84.4%	84.4%	81.2%	81.2%	75.0%
		1	90.6%	81.3%	81.3%	81.2%	75.0%	81.2%
		2	90.6%	81.3%	87.5%	81.2%	81.2%	100.0%
		3	90.6%	81.3%	81.3%	81.2%	81.2%	81.2%
		4	90.6%	81.3%	81.3%	81.2%	81.2%	75.0%
LED-24	24 / 14	0	64.2%	67.9%	67.9%	65.7%	65.3%	65.6%
		1	64.2%	67.2%	67.8%	65.7%	64.6%	67.3%
		2	64.2%	66.6%	64.6%	65.7%	63.0%	64.1%
LED-24	24 / 7	0	64.2%	70.5%	70.5%	65.7%	68.9%	68.9%
		1	64.2%	51.6%	53.0%	65.7%	51.1%	51.7%
		2	64.2%	63.9%	68.5%	65.7%	61.7%	64.2%
Vote	48 / 28	0	91.9%	91.9%	91.9%	97.0%	97.0%	97.0%
		1	91.9%	91.9%	91.9%	97.0%	97.0%	95.6%
		2	91.9%	91.9%	91.9%	97.0%	97.0%	97.0%
Vote	48 / 8	0	91.9%	95.6%	94.8%	97.0%	97.0%	97.0%
		1	91.9%	95.6%	94.8%	97.0%	97.0%	97.0%
		2	91.9%	97.0%	96.3%	97.0%	97.0%	97.0%
DNA	180 / 80	0	93.3%	94.5%	94.9%	92.3%	93.6%	93.4%
		1	93.3%	92.2%	92.5%	92.3%	92.2%	91.2%
		2	93.3%	93.6%	94.4%	92.3%	93.8%	93.4%
DNA	180 / 30	0	93.3%	93.3%	93.8%	92.3%	93.9%	93.8%
		1	93.3%	83.0%	91.2%	92.3%	82.0%	92.7%
		2	93.3%	77.1%	93.6%	92.3%	77.4%	93.4%

Key results:

On Corral (target: $(A \land B) \lor (C \land D)$ , 1 irrelevant feature, 1 correlated feature matching class $75\%$ of the time), backward elimination with $K=2$ successfully removed the correlated and irrelevant features, allowing C4.5 to reach $100.0\%$ accuracy, whereas forward selection always retained the correlated feature.
On LED-24 (7 relevant, 17 irrelevant features independent given class), setting $K=0$ achieved optimal feature selection (70.5% for Naive Bayes, 68.9% for C4.5).
On DNA, reducing features from 180 down to 30 with backward elimination ( $K=2$ ) maintained high accuracy ( $93.6\%$ Naive Bayes, $93.4\%$ C4.5), whereas forward selection degraded to $77.1\%$ and $77.4\%$ .

Knowl 9 — Feature Selection on High-Dimensional Text Categorization

empirical result

The Markov blanket backward elimination algorithm was evaluated on text classification datasets from the Reuters collection:

Reuters1: 1675 word features, 3 classes (coffee, iron-steel, livestock), characterized by distinct vocabularies across topics.
Reuters2: 1646 word features, 3 classes (reserves, gold, gross national product), characterized by overlapping vocabulary across topics.

Eliminating 1000 features reduced the feature space to nearly $1/3$ of its initial size:

Dataset	# Features	$K$	Naive Bayes Accuracy			C4.5 Accuracy
	Orig. / Final		Orig.	Fwd.	Bckwd.	Orig.	Fwd.	Bckwd.
Reuters1	1675 / 675	0	94.2%	95.2%	95.2%	95.2%	96.2%	96.2%
		2	94.2%	91.4%	97.1%	95.2%	88.5%	95.2%
Reuters2	1646 / 646	0	87.5%	87.5%	87.5%	89.8%	93.0%	89.8%
		2	87.5%	88.3%	89.1%	89.8%	91.4%	92.2%

Eliminating 1000 features required approximately $1.5$ hours on a Sun Sparc 10 workstation (compared to thousands of hours estimated for wrapper methods). On Reuters1 with $K=2$ , backward elimination achieved $97.1\%$ accuracy for Naive Bayes and $95.2\%$ for C4.5, outperforming forward selection ( $91.4\%$ and $88.5\%$ , respectively).

Coverage note — No substantial contributed material was omitted from the extraction.

References

1.Almuallim, H. & Dietterich, T. G. (1991), Learning with many irrelevant features, in "Ninth National Conference on Artificial Intelligence", MIT Press, pp. 547–552.
2.Blumer, A., Ehrenfeucht, A., Haussler, D. & Warmuth, M. K. (1987), "Occam's razor", Information Processing Letters 24, 377–380.
3.Caruana, R. & Freitag, D. (1994), Greedy attribute selection, in W. W. Cohen & H. Hirsh, eds, "Machine Learning: Proceedings of the Eleventh International Conference", Morgan Kaufmann Publishers, Inc.
4.Cover, T. M. & Thomas, J. A. (1991), Elements of Information Theory, Wiley.
5.Draper, D. & Hanks, S. (1994), Localized partial evaluation of belief networks, in "Proceedings of the Tenth Annual Conference on Uncertainty in Artificial Intelligence (UAI '94)", pp. 170–177.
6.Duda, R. & Hart, P. (1973), Pattern Classification and Scene Analysis, Wiley.
7.John, G., Kohavi, R. & Pfleger, K. (1994), Irrelevant features and the subset selection problem, in "Machine Learning: Proceedings of the Eleventh International Conference", Morgan Kaufmann, pp. 121–129.
8.Kira, K. & Rendell, L. A. (1992), The feature selection problem: Traditional methods and a new algorithm, in "Tenth National Conference on Artificial Intelligence", MIT Press, pp. 129–134.
9.Kohavi, R. (1995), Wrappers for Performance Enhancement and Oblivious Decision Graphs, PhD thesis, Stanford University, Computer Science department.
10.Kozlov, A. V. & Singh, J. P. (1995), Sensitivities: An alternative to conditional probabilities for bayesian belief networks, in "Proceedings of the Eleventh Annual Conference on Uncertainty in Artificial Intelligence (UAI '95)", pp. 376–385.
11.Kullback, S. & Leibler, R. A. (1951), "On information and sufficiency", Annals of Mathematical Statistics 22, 76–86.
12.Langley, P. & Sage, S. (1994), Induction of selective bayesian classifiers, in "Proceedings of the Tenth Conference on Uncertainty in Artificial Intelligence", Morgan Kaufmann Publishers, Inc., Seattle, WA, pp. 399–406.
13.Langley, P., Iba, W. & Thompson, K. (1992), An analysis of bayesian classifiers, in "Proceedings of the tenth national conference on artificial intelligence", AAAI Press and MIT Press, pp. 223–228.
14.Murphy, P. M. & Aha, D. W. (1995), UCI repository of machine learning databases, http://www.ics.uci.edu/~mlearn/MLRepository.html.
15.Pearl, J. (1988), Probabilistic Reasoning in Intelligent Systems, Morgan Kaufmann, San Mateo, CA.
16.Quinlan, J. R. (1993), C4.5: Programs for Machine Learning, Morgan Kaufmann Publishers, Inc., Los Altos, California.
17.Reuters (1995), Reuters collection available via anonymous ftp., Distribution for research purposes has been granted by Reuters and Carnegie Group. Arrangements for access were made by David Lewis. ftp://ciir-ftp.cs.umass.edu/pub/reuters1.
18.Singh, M. & Provan, G. M. (1996), Efficient learning of selective bayesian network classifiers, Submitted for publication.

Citation

MLA

Koller, D., and M. Sahami. “Toward Optimal Feature Selection”. International Conference on Machine Learning, 1996, pp. 284–92, http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.155.2293.

APA

Koller, D., & Sahami, M. (1996). Toward optimal feature selection. International Conference on Machine Learning, 284–292. http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.155.2293

Chicago

Koller, D., and M. Sahami. 1996. “Toward Optimal Feature Selection”. International Conference on Machine Learning, 284–92. http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.155.2293.

Harvard

Koller, D. and Sahami, M. (1996) “Toward optimal feature selection”, International Conference on Machine Learning, pp. 284–292. Available at: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.155.2293.

Vancouver

1. Koller D, Sahami M (1996) Toward optimal feature selection. International Conference on Machine Learning 284–292

BibTeX

@article{koller1996toward,
  title = {Toward optimal feature selection},
  author = {Koller, Daphne and Sahami, Mehran},
  year = {1996},
  journal = {International Conference on Machine Learning},
  pages = {284-292},
  url = {http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.155.2293}
}

Metadata:DOI registry

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors

Toward Optimal Feature Selection

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Cross-Entropy Criterion for Optimal Feature Subset Selection

Knowl 2 — Markov Blanket Condition for Zero-Divergence Feature Elimination

Knowl 3 — Invariance of Markov Blanket Redundancy Under Successive Feature Removal

Knowl 4 — Approximate Markov Blanket Feature Elimination Algorithm

Knowl 5 — Computational Complexity of Markov Blanket Filter Feature Selection

Knowl 6 — Theoretical Justification of Backward Elimination over Forward Selection