# Feature Selection using Metaheuristics and EAs

URL: https://yarpiz.com/evolutionary-feature-selection-ypml122
Author: Mostapha Kalami Heris
Published: 2015-09-11
Updated: 2026-10-30
Topics: Machine Learning

> Select features for neural-network regression in MATLAB with GA, ACO, SA, PSO, or NSGA-II, including fixed-size and multi-objective formulations.

Feature selection is a common preprocessing task. It seeks a smaller set of model inputs, which can simplify a model and reduce training work. Better generalization is a goal that must be checked on data not used for selection. For more on the concepts and methods, see the [Wikipedia article on feature selection](https://en.wikipedia.org/wiki/Feature_selection).

This post shares a MATLAB implementation of evolutionary feature selection for training feed-forward Artificial Neural Networks (ANNs). A Multi-Layer Perceptron (MLP) solves a nonlinear regression problem: it predicts the body fat percentage from a set of 13 physical measurements. The data file `bodyfat_data.mat` has 252 samples.

## Problem formulations and methods

The project states and solves feature selection in several forms:

- **Selection of a variable number of features, using**
  - *Binary Genetic Algorithm*
- **Selection of a fixed, predetermined number of features** (for example, a subset of 4 features):
  - As a discrete combinatorial optimization problem, using
    - *Ant Colony Optimization (ACO)*
    - *Simulated Annealing (SA)*
  - As a real-valued optimization problem, using
    - *Particle Swarm Optimization (PSO)*
- **Multi-objective feature selection, using**
  - *Non-dominated Sorting Genetic Algorithm II (NSGA-II)*

## How each method encodes a solution

**Binary GA and NSGA-II.** A solution is a binary vector of length 13. A 1 at position $j$ means that feature $j$ is selected, so the number of selected features can change from one solution to another.

**SA and ACO.** A solution is a permutation of the 13 features, and the first $n_f$ features of the permutation are selected. SA creates neighbors with swap, reversion and insertion moves. Each ACO ant builds a permutation step by step, guided by the pheromone matrix.

**PSO.** A particle is a real vector $\mathbf{u} \in [0, 1]^{13}$. Sorting its elements gives a permutation (the random key method), and again the first $n_f$ features are selected. This turns a combinatorial problem into a continuous one that PSO can handle.

## The cost function

For each candidate subset, the code trains a fitting network with 10 hidden neurons (`fitnet`, Levenberg-Marquardt training with `trainlm`, at most 50 epochs) on the selected features. The data is divided at random into 70% training, 15% validation and 15% test samples. The error is the weighted mean square error

$$
E = 0.8\,E_{\text{train}} + 0.2\,E_{\text{test}}
$$

where $E_{\text{test}}$ is measured on the validation and test samples together. Because network training is random, $E$ is averaged over several runs: 3 runs for GA, SA, ACO and PSO, and 5 runs for NSGA-II.

The final cost depends on the formulation:

- **Binary GA:** $z = E\,(1 + \beta\, r_f)$ with $\beta = 0.5$, where $r_f$ is the ratio of selected features. This adds a penalty for using many features.
- **SA, ACO and PSO:** $z = E$, since the number of features $n_f$ is fixed.
- **NSGA-II:** two objectives, the number of selected features $n_f$ and the error $E$, which are minimized together. The result is a Pareto front of subsets that trade size against accuracy.

## What is in the download and how to run it

The download has one folder for each formulation. Each folder has its own copy of the data and of the helper files:

| Folder | Main script | Default settings |
| --- | --- | --- |
| `01 Binary Feature Selection using GA` | `ga.m` | 10 iterations, population 10 |
| `02 Fixed Feature Selection using SA and ACO` | `sa.m`, `aco.m` | $n_f = 4$, 20 iterations (SA with 5 sub-iterations, ACO with 10 ants) |
| `03 Fixed Feature Selection using PSO` | `pso.m` | $n_f = 4$, 20 iterations, swarm of 20 |
| `04 Multi-Objective Feature Selection using NSGA-II` | `nsga2.m` | 20 iterations, population 25 |

To run a method, open its folder in MATLAB and run the main script. Each cost evaluation trains several networks, so a run can take some time even with these small settings. To select a different number of features, change `nf` at the top of `sa.m`, `aco.m` or `pso.m`. The networks are built with `fitnet` and `train`, so the code probably needs the Deep Learning Toolbox (formerly the Neural Network Toolbox).

If you know MATLAB programming, the source code in the download is easy to use in your own research and projects.

## Check the result

The current objective uses validation and test samples while choosing features, so those samples are part of model selection and are not an independent final test. Reserve an untouched holdout, or use nested validation, before reporting predictive accuracy. Keep preprocessing and feature selection inside the training split. Repeated random training reduces some variability but does not remove selection bias. Check the empty-subset case and report the chosen features alongside their count and evaluation protocol.

## Related posts

- [Binary and Real-Coded Genetic Algorithms in MATLAB](https://yarpiz.com/genetic-algorithms-ypea101)
- [Ant Colony Optimization in MATLAB](https://yarpiz.com/ant-colony-optimization-ypea103)
- [Simulated Annealing in MATLAB](https://yarpiz.com/simulated-annealing-ypea105)
- [Particle Swarm Optimization in MATLAB](https://yarpiz.com/particle-swarm-optimization-ypea102)
- [NSGA-II in MATLAB](https://yarpiz.com/nsga2-ypea120)

## How to cite

Mostapha Kalami Heris, Feature Selection using Metaheuristics and EAs (YPML122), Yarpiz, 2015. https://yarpiz.com/evolutionary-feature-selection-ypml122