Have you ever wondered how machine learning algorithms can both categorise data and simplify complex datasets without losing crucial information? Meet Linear Discriminant Analysis (LDA), a powerful technique that does exactly that. Often seen as a foundational method, LDA plays a dual role in both classification and dimensionality reduction.
LDA: More Than Just a Classifier
At its heart, LDA is a generative probabilistic model for classification. What does this mean? It assumes that the data for each class follows a normal (Gaussian) distribution, and critically, that all classes share the same covariance matrix (Σ). This assumption is key to its “linear” nature.
Under these conditions, the Bayes decision rule, which aims for minimum-error-rate classification, simplifies dramatically. It leads directly to a linear discriminant function. This function creates a linear decision boundary that separates the classes in your data. In essence, LDA identifies a straight line (or a hyperplane in higher dimensions) that optimally separates your data points into their respective categories.
The parameters for these discriminant functions, such as the mean vectors (µk) for each class and the common covariance matrix (Σ), are typically learned from your training data using the Maximum Likelihood method. Interestingly, if the true class-conditional probabilities are known, LDA and Logistic Regression yield the same decision rule, though they diverge when parameters are estimated from data.
The Linearity Explained: Where the Magic Happens
The “linear” in Linear Discriminant Analysis isn’t just a label; it’s a mathematical outcome of its core assumption. When the class-conditional Gaussian densities share the same covariance matrix (Σ), the quadratic term xᵀΣ⁻¹x in the discriminant function cancels out when comparing two classes, leaving only terms that are linear in the input features x.
This results in a decision boundary defined by an equation like wᵀx + w₀ = 0. Geometrically, this equation defines a hyperplane in your feature space. The vector w in this equation is special: it is normal (perpendicular) to this decision boundary hyperplane. This is because if you take any two points x₁ and x₂ on the hyperplane, their difference (x₁ - x₂) lies within the hyperplane, and the dot product wᵀ(x₁ - x₂) will be zero. A zero dot product signifies that the vectors are orthogonal.
LDA for Dimensionality Reduction: Finding the Optimal View
Beyond classification, LDA is also a powerful linear dimensionality reduction technique. But it has a crucial difference from other methods like Principal Components Analysis (PCA). While PCA focuses on finding directions that maximise the variance in the data, LDA’s goal is to find a lower-dimensional subspace that maximises the total scatter of the data while keeping the within-class scatter constant.
This is achieved by optimising the Fisher criterion, which is the ratio of the between-class scatter to the within-class scatter. By maximising this ratio, LDA effectively finds projection directions that best separate the different classes.
For a problem with K classes, LDA can project your data onto a subspace with a maximum dimension of K-1. The optimal projection is given by the eigenvectors of a generalised eigenvalue problem (STw = λSWw) corresponding to the largest eigenvalues.
A significant advantage of LDA for dimensionality reduction is that if its underlying assumptions (Gaussian class-conditional densities with identical covariance matrices) hold true, reducing the data to an LDA-derived one-dimensional subspace incurs no loss in classification accuracy for a subsequent classifier. This means all the information relevant for classification is retained, and the resulting classifier remains optimal in the Bayes sense.
Conclusion
Linear Discriminant Analysis stands out as a fundamental tool in machine learning, thanks to its ability to perform robust classification and insightful dimensionality reduction. Its reliance on clear statistical assumptions leads to elegant linear decision boundaries and projections that prioritise class separability, making it an invaluable method for understanding and simplifying complex datasets.