Linear Discriminant Analysis (LDA) in MATLAB

This post shares an open-source MATLAB implementation of Linear Discriminant Analysis (LDA), also called Fisher Discriminant Analysis (FDA). It is used for dimensionality reduction and linear feature extraction from labeled data.

The method goes back to Ronald A. Fisher, who introduced it in 1936 in the same paper that made the iris data set well known. The example in this package uses the iris data too.

The idea

Like Principal Component Analysis (PCA), LDA finds a linear transformation $y = W^T x$ of the data. The difference is the goal. PCA is unsupervised: it looks for the directions with the largest variance and ignores the labels. LDA is supervised: it uses the class labels and looks for the directions that separate the classes best. A direction with a large variance is not always a direction that separates the classes, so the two methods can give very different features.

The Fisher criterion

Assume there are $c$ classes. Class $j$ has $n_j$ samples with the mean $\mu_j$, and $\mu$ is the mean of all the data. LDA uses two scatter matrices.

The within-class scatter measures how spread out each class is around its own mean:

The between-class scatter measures how far the class means are from the overall mean:

A good direction $w$ puts the class means far apart and keeps each class compact. Fisher’s criterion is the ratio of the two:

The directions that maximize $J$ are the solutions of the generalized eigenvalue problem

The eigenvalue $\lambda$ of each direction is its value of $J$, so the directions are sorted by their eigenvalues in descending order. Since $S_B$ is built from $c$ class means, at most $c - 1$ eigenvalues are nonzero. For the iris data, with three classes, there are at most two nonzero between-class discriminant directions.

What is in the package

• LDA.m: the function [Y, W, lambda] = LDA(X, L). It takes the data matrix X (one sample per row) and the class labels L. It returns the transformed data Y, the transformation matrix W and the values lambda.
• main.m: an example script that loads the iris data, calls LDA, and plots each original variable next to the matching transformed variable.
• iris.mat: the iris data used in the example.
• license.txt: the license terms.

Inside LDA.m, the function computes $S_W$ and $S_B$ as above, solves the generalized eigenvalue problem with eig(Sb, Sw), sorts the eigenvectors by their eigenvalues in descending order, and computes Y = X*W. The values in lambda are these sorted eigenvalues, so they show how much each new feature separates the classes.

How to run

Run main.m in MATLAB to see the example. To use the code on your own data, call LDA with your data matrix and a vector of class labels.

[Y, W, lambda] = LDA(X, L);

To reduce the data to $m$ dimensions, keep the first $m$ columns of Y. LDA.m itself uses only core MATLAB functions. The example script main.m converts the iris targets to labels with vec2ind, which probably needs the Deep Learning Toolbox.

Check the result

Fit the projection using training samples and labels only, then apply the same matrix to validation or test samples. Fitting it on all labels before evaluation leaks information from the test set. A singular within-class scatter matrix can occur with redundant features or too few samples, so inspect the eigenproblem rather than assuming every returned direction is stable. The package returns projected features, not a separately trained classifier or a measured classification accuracy.

Applications

LDA is used to reduce the number of features before classification, and also directly as a linear classifier. Well known uses include face recognition (the Fisherfaces method), medical diagnosis from measured variables, and other pattern recognition tasks with labeled data.

Related posts

• Principal Component Analysis (PCA) in Python and MATLAB, the unsupervised counterpart of LDA.