Principal Component Analysis

Chapter 1: Center First — The Preprocessing That Matters

Before PCA can find directions of variation, we must prepare the data with one essential step (and one usually-recommended one). It's tempting to skip, but getting it wrong quietly breaks everything.

Centering: subtract the mean

The essential step is centering: compute the mean of each feature and subtract it, so the data cloud is centered on the origin. Why is this non-negotiable? Because PCA is about variation around the center — the directions the data spreads. If the cloud sits far from the origin, the dominant "direction" PCA finds would just point from the origin toward the cloud's location — capturing where the data is, not how it varies. Centering removes the position so PCA sees only the shape.

Scaling: equalize the units

The usually-recommended step is scaling (standardizing) each feature to unit variance. Here's the trap it avoids: imagine one feature is "weight in grams" (values in the thousands) and another is "fuel economy" (values around 30). The weight feature has vastly larger numbers, so it has vastly larger variance — and PCA, which chases variance, would declare "weight" the most important direction purely because of its units, not its true importance. Scaling each feature to comparable spread prevents arbitrary unit choices from dominating. (Skip scaling only when all features are already in the same meaningful units, like pixel intensities.)

Chapter 2: The Direction of Maximum Variance

Now the heart of PCA, and it's a single beautiful idea: find the direction along which the data varies the most. That direction — the one that captures the largest spread — is the first principal component, the single most informative axis through your data.

To make this precise, we need the idea of projection. Pick any direction — a unit vector u (an arrow of length 1). Now drop each data point perpendicularly onto the line through u; where it lands is the point's projection, a single number measuring how far along u the point sits. Project all the points and you've squashed your 2D cloud down to a 1D set of values on that line. The variance of those projected values — how spread out they are — measures how much of the data's variation that direction captures.

Different directions capture different amounts. Point u along the data's long diagonal, and the projections are widely spread — high variance, lots of information preserved. Point u across the cloud's thin width, and the projections bunch up — low variance, little information. PCA's first component is simply the u that maximizes the projected variance. Find it, and you've found the single direction worth keeping if you could keep only one.

Chapter 3: The Covariance Matrix — All Directions at Once

Rotating a line by hand to find maximum variance works, but it's clumsy and won't scale to 50 dimensions. We need the variance in every direction packaged into one object we can compute with. That object is the covariance matrix — and it's the bridge from "rotate and check" to "solve in one shot."

For centered data, the variance of the projection onto a direction u has a wonderfully compact form. It equals u^TΣu, where Σ (Sigma) is the covariance matrix — the same object from the generative-learning lesson. For 2D data it's a 2×2 matrix: the diagonal entries are the variance of each feature on its own, and the off-diagonals are the covariance — how the two features vary together. Σ encodes the entire shape of the data cloud: its spread along each axis and its tilt.

So "find the direction of maximum variance" becomes the crisp problem: find the unit vector u that maximizes u^TΣu. No more rotating by hand — it's now a clean optimization over a known matrix. The covariance matrix is the data's shape distilled into numbers, and PCA is about to read the best directions straight out of it.

Chapter 4: Eigenvectors — The Punchline

Here's the result that makes PCA one of the most elegant algorithms in all of machine learning. We want the unit vector u that maximizes u^TΣu. When you solve this optimization (with a touch of calculus), the answer is breathtakingly clean: the maximizing direction is the top eigenvector of the covariance matrix Σ.

Let's unpack what that means, building eigenvectors from intuition. An eigenvector of a matrix is a special direction that the matrix only stretches, never rotates — multiply the matrix by that vector and you get the same vector back, just scaled. The amount it's scaled by is the eigenvalue. For a covariance matrix, the eigenvectors are the data's natural axes — the directions along which the cloud is purely stretched — and each eigenvalue is exactly the variance captured along that axis.

So PCA becomes a recipe of stunning simplicity:

• Compute the covariance matrix Σ of your centered data.
• Find its eigenvectors and eigenvalues.
• Sort them by eigenvalue, largest first. The eigenvector with the biggest eigenvalue is the first principal component (most variance); the next is the second; and so on.

That's the entire algorithm. The principal components are the eigenvectors of the covariance, ordered by how much variance they explain. And they're automatically orthogonal (mutually perpendicular), forming a clean new coordinate system perfectly aligned with the data's natural directions. No rotating by hand, no iteration — one eigenvector calculation and you're done.

Chapter 5: The PCA Lab

Let's put the whole pipeline in motion. Below, PCA runs live on data you control: it centers the points, computes the covariance, finds the principal components, and — the payoff — projects the data onto the top component, reducing 2D to 1D. Watch each 2D point collapse onto the principal line, becoming a single number, while losing almost nothing.

(No quiz — the lab is the test. If you can predict, by looking at a cloud's shape, roughly what percent of variance PC1 will explain — near 100% for a thin cloud, near 50% for a round one — you understand what PCA measures and when it helps.)

Chapter 6: Reduce, Reconstruct, and Choose k

PCA gives you a new set of axes (the principal components). The actual reduction is simple: to compress a point, keep only its coordinates along the top k components and drop the rest. A point that lived in d dimensions is now described by just k numbers — its projections onto the most informative directions.

The magic is that you can reconstruct an approximation of the original from those k numbers: rebuild the point as a combination of the kept components. The difference between the original and the reconstruction is the reconstruction error — and it's exactly the variance you threw away by dropping the minor components. This reveals PCA's beautiful dual nature: maximizing the variance you keep is the very same thing as minimizing the reconstruction error. Two views, one algorithm.

Choosing k: the scree plot

How many components should you keep? The same elbow idea from k-means returns, here called a scree plot: plot the variance explained by each component, in order. The early components capture a lot; later ones capture progressively less, until they're just noise. You keep enough components to capture a target fraction of the total variance — 95% or 99% is typical — or you look for the "elbow" where the curve flattens. Often a handful of components capture nearly all the variance of data with hundreds of features — which is exactly why PCA compresses so dramatically.

Chapter 7: PCA in the Wild

PCA isn't just a theoretical tidy-up — it's one of the most-used tools in data science, and the same eigenvector trick powers all of these applications.

Visualization

You can't plot 50-dimensional data, but you can reduce it to its top 2 or 3 components and plot those. Suddenly you can see your data — which cars are similar, which customers cluster together, whether there are distinct groups at all. PCA is the standard first step for laying eyes on high-dimensional data, often paired with clustering (run k-means on the 2D projection).

Compression

Representing each point with k numbers instead of d is literal compression. A famous example: eigenfaces. Each face image of 100×100 pixels is a 10,000-dimensional vector. PCA on a collection of faces finds that just a few dozen "eigenface" directions — combinations of pixels capturing how faces actually vary — reconstruct any face remarkably well. Store ~50 numbers instead of 10,000, and you can still recognize the person. The kept components capture the meaningful variation between faces; the dropped ones were lighting quirks and noise.

Noise reduction & preprocessing

Because noise tends to live in the low-variance directions, dropping those directions cleans the data — reconstructing from the top components alone filters out the noise (this is the RC-pilot "karma" intuition: PCA recovers the true signal from noisy measurements). And as a preprocessing step before supervised learning, reducing dimensions speeds up training and — recalling the bias-variance lesson — shrinks the hypothesis space, helping prevent overfitting on high-dimensional inputs.

Chapter 8: Connections & Cheat Sheet

You've turned "too many redundant numbers" into a precise, one-eigenvector-calculation method for finding the few directions that matter. PCA is the workhorse of dimensionality reduction, and you now understand it from the geometry up.

The whole lesson on one page

PCA in code

python
import numpy as np
def pca(X, k):
    Xc = X - X.mean(0)                 # 1. CENTER (mandatory!)
    Sigma = np.cov(Xc, rowvar=False)        # 2. covariance matrix
    vals, vecs = np.linalg.eigh(Sigma)     # 3. eigenvectors + eigenvalues
    order = np.argsort(vals)[::-1]        # 4. sort by variance, largest first
    U = vecs[:, order[:k]]                 # top-k principal components
    Y = Xc @ U                             # 5. REDUCE: project to k dims
    X_recon = Y @ U.T + X.mean(0)         # reconstruct (approx)
    var_explained = vals[order[:k]].sum() / vals.sum()
    return Y, U, var_explained

# sklearn (uses the numerically-stable SVD under the hood):
from sklearn.decomposition import PCA
p = PCA(n_components=2).fit(X)
print(p.explained_variance_ratio_)      # variance each component captures

Where to go next

• Independent Component Analysis (next) — PCA finds uncorrelated directions of max variance; ICA finds statistically independent sources. The classic use: unmixing overlapping audio (the "cocktail party problem"). A different goal, a different basis.
• Generative Learning & EM/GMM — the same covariance matrix and its eigen-structure power those Gaussian models too.
• Bias & Variance — PCA as a preprocessing step reduces dimensionality and helps control overfitting.
• SVD (Singular Value Decomposition) — the numerically stable way PCA is actually computed, and a powerhouse in its own right (recommender systems, latent semantic analysis).
• Autoencoders — neural networks that learn a nonlinear generalization of PCA; a linear autoencoder recovers PCA exactly.

"The whole is simpler than the sum of its parts." — Willard Gibbs. PCA finds that simpler whole — the few essential directions hiding inside a thicket of correlated measurements.