Linear algebra · 08 · Math for ML · 9 min
MediumRank & the four subspaces
Rank measures how much genuine, independent information a matrix carries — and almost everything interesting about data and models is a statement about rank. Redundancy, compressibility, solvability, and the hidden structure of data are all rank in disguise.
Build the intuition
Rank: independent directions
A matrix's columns are vectors. Some may be redundant — combinations of the others, carrying no new direction. The rank is the number of genuinely independent ones: the true dimensionality of what the matrix spans. A 1000×1000 matrix of rank 3 looks huge but lives in a 3-dimensional shadow — and that gap is exactly what compression and PCA exploit.
Column space, row space, null space — intuitively
The column space is everywhere the matrix can send a vector — all reachable outputs, a flat subspace whose dimension is the rank. The null space is the directions that get crushed to zero — inputs the matrix can't distinguish, the information it destroys. The row space (independent input directions) shares the rank's dimension. Together they answer: what can this transformation reach, and what does it lose?
Real data is secretly low-rank
Pixels in natural images, ratings in a movie matrix, words in documents — all are highly redundant, so their data matrices are nearly low-rank: a few independent patterns explain most of the variation. 'Find the low-rank structure' is the unifying goal of PCA, topic models, and collaborative filtering. The next lesson, SVD, is the tool that finds it.
See it move
Watch 'Squash flat': it collapses 2D space onto a line — rank drops to 1, and the squashed direction becomes the null space (sent to zero, information lost). Full rank keeps space full-dimensional.
A worked example
Spot the hidden rank
A matrix's columns are (1, 2), (2, 4), (3, 6). They look like three columns of data.
But (2,4) = 2·(1,1)... in fact all three are multiples of (1, 2) — one direction, repeated.
Rank = 1. Despite three columns, the matrix carries one independent pattern; its column space is a single line.
Redundancy this stark is rare, but real data is full of near-redundancy — which is why it compresses.
Out in the world
The Netflix matrix
A users×movies ratings matrix is enormous and mostly empty, yet astonishingly low-rank: a few latent 'taste' dimensions (action-lover, indie-fan…) explain most ratings. Recommender systems assume this low rank to fill in the blanks — predicting what you'd rate a film you've never seen. Rank is the reason recommendations work.
Common confusion, cleared
“A bigger matrix carries more information.”
Size and rank are different. A massive low-rank matrix is mostly redundancy — its real content fits in a few independent directions. ML lives off this gap between size and rank.
“The null space is just a technicality.”
It's the information a transformation destroys — why some systems have no unique solution, and why a rank-deficient matrix can't be inverted. The null space is where uniqueness goes to die.
Check yourself
PracticeQuick check
A 10,000 × 10,000 ratings matrix is well-approximated by rank 20. This means…
Recap
- Rank = number of independent directions = true dimensionality.
- Column space = reachable outputs; null space = directions crushed to zero.
- Real data is near-low-rank — the premise of PCA, compression, and recommenders.
Progress saves in this browser.