Analisi dei componenti principali "all'indietro": quanta varianza dei dati è spiegata da una data combinazione lineare delle variabili?

Ho effettuato un'analisi delle componenti principali delle sei variabili , , , , e . Se ho capito bene, PC1 non ruotato mi dice quale combinazione lineare di queste variabili descrive / spiega la maggiore varianza nei dati e PC2 mi dice quale combinazione lineare di queste variabili descrive la successiva più varianza nei dati e così via.$A$$B$$C$$D$$E$$F$

Sono solo curioso - c'è un modo di fare questo "all'indietro"? Diciamo che scelgo una combinazione lineare di queste variabili - ad esempio , potrei capire quanta varianza nei dati che descrive?$A+2B+5C$

Se iniziamo con la premessa che tutte le variabili sono state centrate (pratica standard in PCA), la varianza totale nei dati è solo la somma dei quadrati:

$$T=\sum_{i}(A_{i}^{2}+B_{i}^{2}+C_{i}^{2}+D_{i}^{2}+E_{i}^{2}+F_{i}^{2})$$

Questo è uguale alla traccia della matrice di covarianza delle variabili, che equivale alla somma degli autovalori della matrice di covarianza. Questa è la stessa quantità di cui parla PCA in termini di "spiegazione dei dati" - vale a dire che vuoi che i tuoi PC spieghino la maggior parte degli elementi diagonali della matrice di covarianza. Ora, se rendiamo questa una funzione oggettiva per un insieme di valori previsti in questo modo:

$$S=\sum_{i}\left(\left[A_{i}-\hat{A}_{i}\right]^{2}+\dots+\left[F_{i}-\hat{F}_{i}\right]^{2}\right)$$

Poi le prime principali minimizza elementi $S$ tra tutti rango 1 valori stimati $(\hat{A}_{i},\dots,\hat{F}_{i})$ . Quindi sembrerebbe che la quantità appropriata che stai cercando sia

$$P=1-\frac{S}{T}$$ Per usare il tuo esempio$A+2B+5C$, dobbiamo trasformare questa equazione in previsioni di rango 1. Per prima cosa devi normalizzare i pesi per avere la somma dei quadrati 1. Quindi sostituiamo$(1,2,5,0,0,0)$(somma dei quadrati$30$) con$\left(\frac{1}{\sqrt{30}},\frac{2}{\sqrt{30}},\frac{5}{\sqrt{30}},0,0,0\right)$. Successivamente "segniamo" ogni osservazione secondo i pesi normalizzati:

$$Z_{i}=\frac{1}{\sqrt{30}}A_{i}+\frac{2}{\sqrt{30}}B_{i}+\frac{5}{\sqrt{30}}C_{i}$$

Quindi moltiplichiamo i punteggi per il vettore del peso per ottenere la nostra previsione di grado 1.

$$\begin{pmatrix} \hat{A}_{i} \\ \hat{B}_{i} \\ \hat{C}_{i} \\ \hat{D}_{i} \\ \hat{E}_{i} \\ \hat{F}_{i}\end{pmatrix} =Z_{i}\times\begin{pmatrix} \frac{1}{\sqrt{30}} \\ \frac{2}{\sqrt{30}} \\ \frac{5}{\sqrt{30}} \\ 0 \\ 0 \\ 0\end{pmatrix}$$

Poi inseriamo queste stime in S calcolare P . Puoi anche metterlo nella notazione a matrice, che può suggerire una generalizzazione diversa. Se impostiamo O come matrice N × q dei valori osservati delle variabili ( q = 6 nel tuo caso), ed E come matrice corrispondente di previsioni. Possiamo definire la percentuale di varianza spiegata come:$S$$P$$O$$N\times q$$q=6$$E$

$$\frac{||O||_{2}^{2}-||O-E||_{2}^{2}}{||O||_{2}^{2}}$$

Dove | | . | | 2 è la norma della matrice di Frobenius . Quindi potresti "generalizzare" questo come un altro tipo di norma matriciale e otterrai una misura della differenza di "variazione spiegata", sebbene non sia "varianza" di per sé a meno che non sia una somma di quadrati.$||.||_{2}$

Let's say I choose some linear combination of these variables -- e.g. $A+2B+5C$, could I work out how much variance in the data this describes?

This question can be understood in two different ways, leading to two different answers.

A linear combination corresponds to a vector, which in your example is $[1, 2, 5, 0, 0, 0]$. This vector, in turn, defines an axis in the 6D space of the original variables. What you are asking is, how much variance does projection on this axis "describe"? The answer is given via the notion of "reconstruction" of original data from this projection, and measuring the reconstruction error (see Wikipedia on Fraction of variance unexplained). Turns out, this reconstruction can be reasonably done in two different ways, yielding two different answers.

---

Approach #1

Let $\newcommand{\S}{\boldsymbol \Sigma} \newcommand{\w}{\mathbf w} \newcommand{\v}{\mathbf v}\newcommand{\X}{\mathbf X} \X$ be the centered dataset ($n$ rows correspond to samples, $d$ columns correspond to variables), let $\S$ be its covariance matrix, and let $\w$ be a unit vector from $\mathbb R^d$. The total variance of the dataset is the sum of all $d$ variances, i.e. the trace of the covariance matrix: $T = \mathrm{tr}(\S)$. The question is: what proportion of $T$ does $\w$ describe? The two answers given by @todddeluca and @probabilityislogic are both equivalent to the following: compute projection $\X \w$, compute its variance and divide by $T$:

$$R^2_\mathrm{first} = \frac{\mathrm{Var}(\X \w)}{T} = \frac{\w^\top \S \w}{\mathrm{tr}(\S)}.$$

This might not be immediately obvious, because e.g. @probabilityislogic suggests to consider the reconstruction $\X \w \w^\top$ and then to compute

$$\frac{\|\X\|^2 - \|\X-\X \w \w^\top\|^2}{\|\X\|^2},$$ but with a little algebra this can be shown to be an equivalent expression.

---

Approach #2

Okay. Now consider a following example: $\X$ is a $d=2$ dataset with covariance matrix

$$\S = \left(\begin{array}{c}1&0.99\\0.99&1\end{array}\right)$$ and $\mathbf w = (\begin{array}{}1&0\end{array})^\top$ is simply an $x$ vector:

The total variance is $T=2$. The variance of the projection onto $\w$ (shown in red dots) is equal to $1$. So according to the above logic, the explained variance is equal to $1/2$. And in some sense it is: red dots ("reconstruction") are far away from the corresponding blue dots, so a lot of the variance is "lost".

On the other hand, the two variables have $0.99$ correlation and so are almost identical; saying that one of them describes only $50\%$ of the total variance is weird, because each of them contains "almost all the information" about the second one. We can formalize it as follows: given projection $\X\w$, find a best possible reconstruction $\X\w\v^\top$ with $\v$ not necessarily the same as $\w$, and then compute the reconstruction error and plug it into the expression for the proportion of explained variance:

$$R^2_\mathrm{second}=\frac{\|\X\|^2 - \|\X-\X \w \v^\top\|^2}{\|\X\|^2},$$ where $\v$ is chosen such that $\|\X-\X \w \v^\top\|^2$ is minimal (i.e. $R^2$ is maximal). This is exactly equivalent to computing $R^2$ of multivariate regression predicting original dataset $\X$ from the $1$-dimensional projection $\X\w$.

It is a matter of straightforward algebra to use regression solution for $\v$ to find that the whole expression simplifies to

$$R^2_\mathrm{second}=\frac{\|\S \w\|^2}{\w^\top \S \w \cdot \mathrm{tr}(\S)}.$$ In the example above this is equal to $0.9901$, which seems reasonable.

Note that if (and only if) $\w$ is one of the eigenvectors of $\S$, i.e. one of the principal axes, with eigenvalue $\lambda$ (so that $\S \w = \lambda \w$), then both approaches to compute $R^2$ coincide and reduce to the familiar PCA expression

$$R^2_\mathrm{PCA} = R^2_\mathrm{first} = R^2_\mathrm{second} = \lambda/\mathrm{tr}(\S) = \lambda/\sum \lambda_i.$$

PS. See my answer here for an application of the derived formula to the special case of $\w$ being one of the basis vectors: Variance of the data explained by a single variable.

---

Appendix. Derivation of the formula for $R^2_\mathrm{second}$

Finding $\v$ minimizing the reconstruction $\|\X-\X \w \v^\top\|^2$ is a regression problem (with $\X \w$ as univariate predictor and $\X$ as multivariate response). Its solution is given by

$$\v^\top = \left((\X \w)^\top (\X \w)\right)^{-1}(\X \w)^\top \X = (\w^\top \S \w)^{-1} \w^\top \S.$$

Next, the $R^2$ formula can be simplified as

$$R^2=\frac{\|\X\|^2 - \|\X-\X \w \v^\top\|^2}{\|\X\|^2} = \frac{\|\X \w \v^\top\|^2}{\|\X\|^2}$$ due to the Pythagoras theorem, because the hat matrix in regression is an orthogonal projection (but it is also easy to show directly).

Plugging now the equation for $\v$, we obtain for the numerator:

$$\|\X \w \v^\top\|^2 = \mathrm{tr}\left(\X \w \v^\top (\X \w \v^\top)^\top\right) = \mathrm{tr}(\X\w\w^\top\S\S\w\w^\top\X^\top)/(\w^\top\S\w)^2=\mathrm{tr}(\w^\top\S\S\w)/(\w^\top\S\w) = \|\S\w\|^2 / (\w^\top\S\w).$$

The denominator is equal to $\|\X\|^2 = \mathrm{tr}(\S)$ resulting in the formula given above.

Let the total variance, $T$, in a data set of vectors be the sum of squared errors (SSE) between the vectors in the data set and the mean vector of the data set,

$$T = \sum_{i} (x_i-\bar{x}) \cdot (x_i-\bar{x})$$ where $\bar{x}$ is the mean vector of the data set, $x_i$ is the ith vector in the data set, and $\cdot$ is the dot product of two vectors. Said another way, the total variance is the SSE between each $x_i$ and its predicted value, $f(x_i)$, when we set $f(x_i)=\bar{x}$.

Now let the predictor of $x_i$, $f(x_i)$, be the projection of vector $x_i$ onto a unit vector $c$.

$$f_c(x_i) = (c \cdot x_i)c$$

Then the $SSE$ for a given $c$ is

$$SSE_c = \sum_i (x_i - f_c(x_i)) \cdot (x_i - f_c(x_i))$$

I think that if you choose $c$ to minimize $SSE_c$, then $c$ is the first principal component.

If instead you choose $c$ to be the normalized version of the vector $(1, 2, 5, ...)$, then $T-SSE_c$ is the variance in the data described by using $c$ as a predictor.