Eigenvectors, SVD, and PCA
Singular value decomposition (SVD) and principal components analysis (PCA) are two widely used methods for matrix factorization and dimensionality reduction. I explain both in detail, both formally and with code, and show how PCA relies on SVD.
Introduction
Algos: http://www.math.pitt.edu/~sussmanm/2071Spring08/lab09/index.html
Intuition and examples: https://math.stackexchange.com/questions/92171/what-do-eigenvalues-have-to-do-with-pictures/92187#92187
Prior to this writing, whenever I am discussed singular value decomposition (SVD) or principal components analysis (PCA), my explanations are hand-wavy. SVD, I know, decomposes a matrix $A$ into three matrices:
Where $U$ and $V$ are orthogonal matrices and $\Sigma$ is a diagonal matrix. But what does this really mean? How do you compute these matrices? What are we doing to our data? And PCA finds linear projections s.t. the lower-dimensional representation has maximum variance. Again, what does that mean and how can we formalize it?
There is also a relationship between SVD and PCA that is important. To actually implement PCA in code, you can just call a function that computes the SVD. The goal of this post is to concretize what SVD and PCA are doing. As always, my gut-checks are words and code. If I cannot explain it and cannot write it down computationally, my understanding is limited in some way.
Notes
orthogonality, the meaning of matrix multiplication, etc. bianca claims you can only rotate or skew a vector with a matrix—directly implies no nonlinear transformation and PCA Diana: “yeah i svd is trickier, but pca you can just call SVD (which i think is a good thing to do)”