Projection Matrices Transform Whole Datasets

A projection can explain one observation with a single direction. The more useful question for an AI pipeline is what happens when the observation is a dataset: can one small linear rule transform every row consistently?

Yes. A projection matrix applies the same nearest-point operation to every sample. It preserves the component along a chosen direction and removes the orthogonal component, so a point cloud becomes a one-direction representation while every output still has the original number of coordinates.

By the end of this lesson, you will be able to construct a line-projection matrix, apply it to a row-stored dataset, and check the two identities that make it a projection: symmetry and idempotence.

Learning outcome: Construct and check a simple projection matrix.

This lesson builds on Lesson 035: Matrices Represent Linear Mappings, where matrices became reusable rules, and Lesson 050: Projection Finds the Nearest Point, where one vector was projected onto a line. Section 3.8 of the official Mathematics for Machine Learning book develops the projection-matrix construction. The dataset, calculations, visual, NumPy implementation, and assessments here are original.

Turn one line projection into a matrix

Let $b\in\mathbb{R}^d$ be a nonzero direction vector, and let $U=\operatorname{span}\{b\}$ be the line through the origin that it defines. For one column-vector observation $x\in\mathbb{R}^d$, the previous lesson gave

$$\operatorname{proj}_{U}(x)=\frac{x^{\mathsf T}b}{b^{\mathsf T}b}b.$$

The scalar $x^{\mathsf T}b$ is a dot product. Rewrite the right side so that $x$ appears as the final factor:

$$\frac{x^{\mathsf T}b}{b^{\mathsf T}b}b =\frac{bb^{\mathsf T}}{b^{\mathsf T}b}x.$$

That rearrangement exposes a reusable linear rule.

The matrix has two immediate geometric consequences. First,

$$P^{\mathsf T} =\left(\frac{bb^{\mathsf T}}{b^{\mathsf T}b}\right)^{\mathsf T} =\frac{bb^{\mathsf T}}{b^{\mathsf T}b}=P,$$

so $P$ is symmetric. Second, once a vector has reached the line, projecting it again does nothing:

$$P^2=P.$$

This is called idempotence. To see why it holds, let $c=b^{\mathsf T}b$. Then

$$P^2=\frac{bb^{\mathsf T}}{c}\frac{bb^{\mathsf T}}{c} =\frac{b(b^{\mathsf T}b)b^{\mathsf T}}{c^2} =\frac{bc b^{\mathsf T}}{c^2}=P.$$

Symmetry describes the right-angle geometry; idempotence describes the fact that the output is already settled on the target line. Neither property says that $P$ is invertible. A one-direction projection usually discards the other $d-1$ directions.

Apply the same rule to a row-stored dataset

AI data is often stored with one sample per row. Let

$$X=\begin{bmatrix} x_1^{\mathsf T}\\ x_2^{\mathsf T}\\ \vdots\\ x_m^{\mathsf T} \end{bmatrix}\in\mathbb{R}^{m\times d},$$

where $m$ is the number of samples and $d$ is the number of features. Because $P$ is symmetric, multiplying on the right applies the column-vector projection to each row’s transpose:

$$Y=XP =\begin{bmatrix} x_1^{\mathsf T}P\\ x_2^{\mathsf T}P\\ \vdots\\ x_m^{\mathsf T}P \end{bmatrix}.$$

Each row of $Y$ is still length $d$, but every row lies on the one-dimensional line $U$. The data has not been reduced to one coordinate yet. The coefficient matrix

$$z=\frac{Xb}{b^{\mathsf T}b}\in\mathbb{R}^{m}$$

stores the one coordinate per sample, and the reconstruction is $Y=z b^{\mathsf T}$. Keeping $Y$ is convenient when the downstream code expects the original feature width; keeping $z$ is convenient when the goal is a compact one-number representation.

A four-sample point cloud, transformed two ways

Use the direction

$$b=\begin{bmatrix}2\\1\end{bmatrix}, \qquad b^{\mathsf T}b=2^2+1^2=5.$$

The outer product and projection matrix are

$$bb^{\mathsf T} =\begin{bmatrix}2\\1\end{bmatrix}\begin{bmatrix}2&1\end{bmatrix} =\begin{bmatrix}4&2\\2&1\end{bmatrix}, \qquad P=\frac15\begin{bmatrix}4&2\\2&1\end{bmatrix} =\begin{bmatrix}0.8&0.4\\0.4&0.2\end{bmatrix}.$$

Take four two-feature observations as rows:

$$X=\begin{bmatrix} 3&1\\ 1&3\\ -2&-1\\ 0&4 \end{bmatrix}.$$

Geometry: each row moves to U

• Original observation
• Projected observation
• Target line U

Matrix: one rule for all rows

P =154  22  1

With samples stored as rows, multiply X on the right by P.

Y = X @ P

Exact numerical fallback for the plotted point cloud.

Sample	Original xᵢ	Coefficient zᵢ	Projected yᵢ	Residual xᵢ − yᵢ
x₁	(3, 1)	1.4	(2.8, 1.4)	(0.2, −0.4)
x₂	(1, 3)	1	(2, 1)	(−1, 2)
x₃	(−2, −1)	−1	(−2, −1)	(0, 0)
x₄	(0, 4)	0.8	(1.6, 0.8)	(−1.6, 3.2)

The point plot shows the geometry; the table preserves the values without relying on color, scale, or an interactive view.

The visual exposes a useful distinction. The matrix is a rule for every input, not a lookup table for these four points. Any new row in the same feature space can be multiplied by the same $P$, and points already on $U$ remain unchanged.

Check the matrix with NumPy

NumPy’s outer computes the outer product of two vectors, which constructs the numerator $bb^{\mathsf T}$. Its matmul function performs matrix products; the @ operator is its array shorthand. We can therefore write the row-stored transformation as X @ P and use linalg.norm with axis=1 to measure each row’s residual length.

allclose checks numerical equality within chosen tolerances. The checks below do not replace the algebra: they catch an implementation mistake while acknowledging that floating-point arithmetic may leave a tiny remainder instead of literal zero.

import numpy as np

b = np.array([2.0, 1.0])
X = np.array(
    [
        [3.0, 1.0],
        [1.0, 3.0],
        [-2.0, -1.0],
        [0.0, 4.0],
    ]
)

P = np.outer(b, b) / np.dot(b, b)
Y = X @ P
residuals = X - Y
residual_lengths = np.linalg.norm(residuals, axis=1)

tolerance = 1e-12
symmetric = np.allclose(P.T, P, rtol=0.0, atol=tolerance)
idempotent = np.allclose(P @ P, P, rtol=0.0, atol=tolerance)
already_projected = np.allclose(Y @ P, Y, rtol=0.0, atol=tolerance)

print(P)
print(Y)
print(residual_lengths)
print(symmetric, idempotent, already_projected)

Verified output:

[[0.8 0.4]
 [0.4 0.2]]
[[ 2.8  1.4]
 [ 2.   1. ]
 [-2.  -1. ]
 [ 1.6  0.8]]
[0.4472136  2.23606798 0.         3.57770876]
True True True

The three Boolean results answer different questions. symmetric checks the matrix’s transpose identity, idempotent checks $P^2=P$, and already_projected checks that the entire transformed dataset stays fixed on a second pass. The residual lengths show that a point can lie on the target line with zero error, while another point can retain a substantial orthogonal component.

What the matrix keeps—and what it cannot keep

For any vector $x$, the output $Px$ lies on $U$. The coefficient $z=(x^{\mathsf T}b)/(b^{\mathsf T}b)$ is the part of $x$ that the chosen line can explain, while $x-Px$ is perpendicular to $b$. The projection is therefore a one-dimensional model of a $d$-dimensional observation.

This is the basic shape of dimensionality reduction: choose a lower-dimensional feature space, preserve the nearest point in that space, and measure the reconstruction error that remains. In an embedding or feature pipeline, a direction could represent a learned pattern and $z$ could be retained as its strength. The projection matrix can apply that same representation rule to a batch, but the choice of direction still determines what information is preserved. A small residual means “close under this Euclidean line,” not “the sample’s meaning has been completely captured.”

Boundary cases and layout traps

A zero direction is not a line. If $b=0$, then $b^{\mathsf T}b=0$ and the formula divides by zero. A production function should reject that input rather than produce nan values or silently call the zero vector a direction.

An offset line needs centering. The matrix above projects onto a line through the origin. For an affine line $c+tb$, first project $x-c$ and then add $c$:

$$c+\frac{(x-c)^{\mathsf T}b}{b^{\mathsf T}b}b.$$

Applying $P$ directly to $x$ would use the wrong line. For a whole dataset, subtract $c$ from each row before the matrix multiplication and add it back afterward.

Do not let storage orientation choose the mathematics for you. For a row matrix $X\in\mathbb{R}^{m\times d}$ use $XP$; for a column matrix use $PX_{\mathrm{col}}$. A transposed dataset can make a wrong multiplication look plausible while swapping “number of samples” and “number of features.” Check the shapes before interpreting the output.

The single-vector projection from Lesson 050 becomes a batch operation without changing its geometry. Lesson 052 will use the same nearest-point idea for a subspace generated by several columns, where the projection matrix can retain more than one learned direction.

Check your understanding

Question 1

For b = (1, 2), construct the projection matrix P = bbᵀ/(bᵀb) and state its shape.

Show the step-by-step solution

First compute the denominator:

$$b^{\mathsf T}b=1^2+2^2=5.$$

Then form the outer product, placing $b$ in the left factor and $b^{\mathsf T}$ in the top factor:

$$bb^{\mathsf T} =\begin{bmatrix}1\\2\end{bmatrix}\begin{bmatrix}1&2\end{bmatrix} =\begin{bmatrix}1&2\\2&4\end{bmatrix}.$$

Therefore

$$P=\frac15\begin{bmatrix}1&2\\2&4\end{bmatrix} =\begin{bmatrix}0.2&0.4\\0.4&0.8\end{bmatrix}.$$

Because $b$ has two coordinates, $P$ is a $2\times2$ matrix.

Question 2

Using b = (2, 1) and x = (1, 3), calculate the projection coefficient and projected point.

Show the step-by-step solution

Compute the dot product and squared length:

$$x^{\mathsf T}b=1\cdot2+3\cdot1=5, \qquad b^{\mathsf T}b=2^2+1^2=5.$$

The coefficient is $z=5/5=1$. Multiply it by the direction:

$$\operatorname{proj}_{U}(x)=zb=1(2,1)=(2,1).$$

The result lies on $U$ because it is a scalar multiple of $b$.

Question 3

A dataset X has four rows and two feature columns. Should its row-wise projection onto P be X @ P or P @ X? Explain the dimensions.

Show the step-by-step solution

The shape of $X$ is $4\times2$, and a projection matrix for two-dimensional features has shape $2\times2$. The product

$$XP:(4\times2)(2\times2)\longrightarrow4\times2$$

is defined and keeps one output row for each sample. Thus the row-wise operation is X @ P. The product $PX$ would try to multiply a $2\times2$ matrix by a $4\times2$ matrix; the inner dimensions $2$ and $4$ do not match. If samples were stored as columns in a $2\times4$ matrix, then $P$ would act on the left instead.

Question 4

Verify idempotence for P = [[0.8, 0.4], [0.4, 0.2]] by computing P². What does the result mean for a projected dataset?

Show the step-by-step solution

Multiply the matrix by itself:

$$P^2 =\begin{bmatrix}0.8&0.4\\0.4&0.2\end{bmatrix} \begin{bmatrix}0.8&0.4\\0.4&0.2\end{bmatrix} =\begin{bmatrix} 0.64+0.16&0.32+0.08\\ 0.32+0.08&0.16+0.04 \end{bmatrix} =\begin{bmatrix}0.8&0.4\\0.4&0.2\end{bmatrix}=P.$$

Therefore $(XP)P=X(P^2)=XP$. A second pass leaves every projected row unchanged; it does not continue shrinking the dataset.

Question 5

Why is the line-projection matrix symmetric, and how does that relate to the residual?

Show the step-by-step solution

The matrix is

$$P=\frac{bb^{\mathsf T}}{b^{\mathsf T}b}.$$

The denominator is a scalar, and transposing the numerator reverses the factors:

$$(bb^{\mathsf T})^{\mathsf T}=bb^{\mathsf T}.$$

So $P^{\mathsf T}=P$. Geometrically, the projection uses the usual Euclidean inner product, so the difference $x-Px$ has zero dot product with $b$. The symmetry is an algebraic signature of that orthogonal, right-angle projection; it does not by itself imply that every vector is preserved.

Question 6

An engineer says, "Because P has two rows and two columns, it keeps two independent directions." Correct the claim for a nonzero two-dimensional direction b.

Show the step-by-step solution

The shape of a matrix is not its number of independent directions. Both columns of $P=bb^{\mathsf T}/(b^{\mathsf T}b)$ are scalar multiples of $b$, so its columns span only the one-dimensional line $U$. Any component perpendicular to $b$ is removed. Thus a $2\times2$ projection matrix can have rank one: it keeps one direction while using two coordinates to store each output point.

Question 7

What should a NumPy implementation do when b = [0.0, 0.0], and why is adding a tiny denominator not a faithful fix?

Show the step-by-step solution

The zero vector does not define a line, and its denominator is $b^{\mathsf T}b=0$. The implementation should reject it explicitly, for example with a check that the squared norm is greater than zero, and explain the invalid input. Adding an arbitrary tiny number to the denominator would manufacture a very large or unstable matrix for an undefined geometric task. A numerical regularizer may be appropriate for a different, explicitly defined problem, but it does not turn the zero vector into a direction.

Question 8

For an affine line c + tb, why can’t the origin-line matrix P be applied directly to x? Give the corrected row-wise dataset operation.

Show the step-by-step solution

The matrix $P$ maps to a line through the origin. An affine line has offset $c$, so the relevant vector is $x-c$, not $x$. For each row of a dataset, center first, project, then restore the offset:

$$Y=(X-\mathbf{1}c^{\mathsf T})P+\mathbf{1}c^{\mathsf T},$$

where $\mathbf{1}$ is an $m$-entry column of ones. The subtraction and addition are row-wise. Applying $XP$ without centering would project onto the parallel line through zero and answer a different question.