Projection Finds the Nearest Point

An AI system often has to explain an observation with fewer directions than the observation itself contains. If a model subspace is a line, which point on that line is the best explanation of a vector that does not land on it exactly?

An orthogonal projection answers that question. It keeps the part of the observation that points along the model direction and leaves a residual at a right angle. That residual is not a vague leftover: its length is the smallest possible Euclidean error among all points on the line.

By the end of this lesson, you will be able to compute the projection of a vector onto a line, prove why its residual is perpendicular, compare it with other candidate points, and implement the calculation in NumPy without forming an explicit matrix inverse.

Learning outcome: Compute projection onto a line and explain the perpendicular residual.

This lesson connects Lesson 046: Orthonormal Bases Simplify Coordinates, where inner products read coordinates in a friendly basis, with Lesson 048: Orthogonal Complements Capture What Is Left, where a residual was identified as a direction outside a subspace. The official Mathematics for Machine Learning book maps this topic to Section 3.8. The line, numbers, visual, implementation, AI connection, and assessments here are original.

A line gives the model a one-direction family

Let $b\in\mathbb{R}^n$ be a nonzero direction vector. The line through the origin with that direction is the one-dimensional subspace

$$U=\operatorname{span}\{b\}=\{tb:t\in\mathbb{R}\}.$$

The scalar $t$ is a coordinate along the line. Every value of $t$ gives one candidate point, so projecting an observation $x\in\mathbb{R}^n$ onto $U$ means choosing the coordinate that makes $tb$ as close to $x$ as possible.

The condition $r^{\mathsf T}b=0$ is enough to find $\alpha$. Substitute $p=\alpha b$ and expand the dot product:

$$\begin{aligned} 0&=(x-\alpha b)^{\mathsf T}b\\ &=x^{\mathsf T}b-\alpha b^{\mathsf T}b. \end{aligned}$$

Because $b$ is nonzero, $b^{\mathsf T}b=\lVert b\rVert_2^2$ is positive. We can divide by it and obtain the projection coefficient

$$\boxed{\alpha=\frac{x^{\mathsf T}b}{b^{\mathsf T}b}}.$$

The projected point and residual then follow by substitution:

$$\boxed{p=\frac{x^{\mathsf T}b}{b^{\mathsf T}b}\,b,\qquad r=x-p.}$$

If $b$ has unit length, the denominator is $1$, which is the shortcut from an orthonormal coordinate system. For an arbitrary direction, omitting $b^{\mathsf T}b$ silently assumes that $b$ is a unit vector.

Why the perpendicular point is the nearest point

The dot-product check tells us that the residual is perpendicular, but it also proves the nearest-point claim. Take any other point on the line, $tb$. Since $p=\alpha b$,

$$x-tb=(x-p)+(p-tb)=r+(\alpha-t)b.$$

The two terms are perpendicular: $r$ is perpendicular to $b$, so it is also perpendicular to the scaled vector $(\alpha-t)b$. The Pythagorean theorem now gives

$$\lVert x-tb\rVert_2^2 =\lVert r\rVert_2^2+(\alpha-t)^2\lVert b\rVert_2^2.$$

The second term is never negative, and it is $0$ exactly when $t=\alpha$. Therefore $p=\alpha b$ is the unique nearest point on the line. This is why the residual length $\lVert r\rVert_2$ is also called the projection error or reconstruction error for this one-direction model.

A worked projection with a numerical nearest-point check

Use an observation and model direction that are deliberately not aligned:

$$x=\begin{bmatrix}3\\1\end{bmatrix}, \qquad b=\begin{bmatrix}2\\1\end{bmatrix}.$$

The value $1.4$ wins even when we compare it with nearby candidate coordinates:

Coordinate $t$	Candidate point $tb$	Distance $\lVert x-tb\rVert_2$
$0$	$(0,0)$	$\sqrt{10}\approx3.162$
$1$	$(2,1)$	$1$
$1.4$	$(2.8,1.4)$	$\sqrt{5}/5\approx0.447$
$2$	$(4,2)$	$\sqrt{2}\approx1.414$

The candidate at $t=1$ is closer than the one at $t=2$ because $1$ is nearer to the optimal coordinate $1.4$ along the line. The line has infinitely many candidates, so a small comparison table illustrates the theorem but does not replace the formula.

Choose the observation coordinates

The line direction is fixed at b = (2, 1). Move x with the sliders, then compare the projected point and residual in the table.

Observation x₁

3

Observation x₂

1

Reset to x = (3, 1)

Keyboard access: focus the blue point and use Arrow keys to move it by 0.25. Hold Shift for a move of 1.

x = (3, 1); α = 1.4; p = (2.8, 1.4); r = (0.2, −0.4); r · b = 0; distance = 0.45.

Current projection values, also available without the plot.

Quantity	Value	Meaning
Observation x	(3, 1)	The reader-controlled vector being approximated.
Coefficient α	1.4	The coordinate along the model direction.
Projected point p	(2.8, 1.4)	The closest point on the line U.
Residual r	(0.2, −0.4)	The part left after the line explains its share.
Residual · direction b	0	Zero within the stated floating-point tolerance.
Residual length	0.45	The Euclidean projection error.

Distances to a few candidate points on U.

Coordinate t	Candidate point tb	Distance to x
0	(0, 0)	3.16
1	(2, 1)	1
1.4 (nearest for this x)	(2.8, 1.4)	0.45
2	(4, 2)	1.41

The tables are the complete numerical fallback. Moving x changes every value, but the geometric rule remains: the residual is perpendicular and the projection minimizes distance.

The blue observation can move with the pointer, coordinate sliders, or arrow keys. The green point is its projection onto the dashed line; the amber segment is the perpendicular residual.

A transparent NumPy implementation

The implementation should mirror the derivation. numpy.dot computes the one-dimensional inner products, while numpy.linalg.norm computes the Euclidean lengths. The code below also compares the reconstruction with the original observation and checks the perpendicular dot product using an explicit tolerance. numpy.allclose compares arrays within absolute and relative tolerances; choosing the tolerance is part of a numerical test, not a way to prove exact equality in floating-point arithmetic.

import numpy as np

x = np.array([3.0, 1.0])
b = np.array([2.0, 1.0])

coefficient = np.dot(x, b) / np.dot(b, b)
projected = coefficient * b
residual = x - projected

tolerance = 1e-12
reconstruction_ok = np.allclose(
    projected + residual, x, rtol=0.0, atol=tolerance
)
orthogonality_ok = np.isclose(
    np.dot(residual, b), 0.0, rtol=0.0, atol=tolerance
)

candidates = np.array([0.0, 1.0, 1.4, 2.0])
candidate_points = candidates[:, None] * b
candidate_distances = np.linalg.norm(x - candidate_points, axis=1)
best_candidate = candidates[np.argmin(candidate_distances)]

print(coefficient)
print(projected)
print(residual)
print(np.dot(residual, b))
print(np.linalg.norm(residual))
print(reconstruction_ok, orthogonality_ok)
print(candidate_distances)
print(best_candidate)

Verified output:

1.4
[2.8 1.4]
[ 0.2 -0.4]
4.440892098500626e-16
0.4472135954999579
True True
[3.16227766 1.         0.4472136  1.41421356]
1.4

There is no explicit inverse in this line projection. The only division is by the scalar $b^{\mathsf T}b$, which is positive because the direction is nonzero. For a higher-dimensional model subspace, several coefficients become coupled and the implementation needs a stable linear-algebra solve; the next lessons will make that extension explicit.

One model direction can explain one AI feature pattern

Suppose an encoder produces a two-feature observation $x$ and a domain expert has identified one meaningful pattern $b$. The line $U=\operatorname{span}\{b\}$ contains every possible strength and sign of that pattern. The coefficient

$$\alpha=\frac{x^{\mathsf T}b}{b^{\mathsf T}b}$$

is the best one-number summary of $x$ using that direction. Reconstructing $\alpha b$ gives the model’s explained component; $r=x-\alpha b$ is what that one-direction model cannot explain. The orthogonality check says the residual contains no additional component along the same direction, not that it contains no information at all.

This is the geometric core of a one-feature least-squares fit. If the feature direction is normalized, the denominator disappears. If the chosen feature is not normalized, keeping the denominator prevents its arbitrary scale from changing the reconstructed point. In a real representation pipeline, the choice of direction and the metric still need validation: a low residual means “close under this line and Euclidean distance,” not “semantically complete.”

Boundary cases that change the answer

A zero direction is not a line. If $b=0$, then $b^{\mathsf T}b=0$ and the coefficient formula divides by zero. There is no direction to project onto; a production implementation should reject this input rather than manufacture a coefficient.

An affine line has an offset. The formula above is for a line through the origin. For $c+tb$, first center the observation: project $x-c$ onto $b$, then add $c$ back:

$$\operatorname{proj}_{c+\operatorname{span}\{b\}}(x) =c+\frac{(x-c)^{\mathsf T}b}{b^{\mathsf T}b}b.$$

Applying the origin-line formula directly to $x$ would force the answer onto a different line.

Orthogonal does not mean irrelevant. A residual can be perpendicular to the chosen model direction and still carry a large signal, a missing feature, or a systematic error. The projection only answers the question that the selected subspace and distance define.

The projection turns “which explanation is closest?” into a reproducible coefficient, point, residual, and tolerance check. Lesson 051 will lift this single-observation operation into a projection matrix that can transform a whole dataset, while Lesson 052 will connect the same nearest-point geometry to least-squares regression.

Check your understanding

Question 1

For x = (4, 1) and b = (1, 1), compute the projection coefficient α and the projected point.

Show the step-by-step solution

First compute the two dot products:

$$x^{\mathsf T}b=4\cdot1+1\cdot1=5, \qquad b^{\mathsf T}b=1^2+1^2=2.$$

Therefore $\alpha=5/2$. Multiply this scalar by $b$:

$$p=\frac52(1,1)=\left(\frac52,\frac52\right).$$

The result lies on the line $y=x$, as every multiple of $(1,1)$ must.

Question 2

For the same x = (4, 1) and b = (1, 1), compute the residual and verify that it is perpendicular to b.

Show the step-by-step solution

From the previous calculation,

$$r=x-p=(4,1)-\left(\frac52,\frac52\right) =\left(\frac32,-\frac32\right).$$

Now take the dot product with the line direction:

$$r^{\mathsf T}b=\frac32\cdot1+\left(-\frac32\right)\cdot1=0.$$

The zero confirms the residual is orthogonal to the line direction under the standard dot product.

Question 3

Why does the projection coefficient include bᵀb in the denominator? What would go wrong if b were replaced by 3b but the denominator were omitted?

Show the step-by-step solution

The coefficient is obtained from the orthogonality equation

$$0=(x-\alpha b)^{\mathsf T}b=x^{\mathsf T}b-\alpha b^{\mathsf T}b.$$

Solving for $\alpha$ requires dividing by $b^{\mathsf T}b$, which accounts for the length of the direction. If $b$ is replaced by $3b$, then the numerator $x^{\mathsf T}(3b)$ is also multiplied by $3$, while the correct denominator is multiplied by $9$. The ratio therefore becomes one-third as large, and the final product $\alpha(3b)$ stays unchanged. Omitting the denominator would make the result depend on the arbitrary scale used to write the same line.

Question 4

Show why the projection is closer than an arbitrary candidate tb without checking a long list of candidate points.

Show the step-by-step solution

Write the observation as $x=p+r$ with $p=\alpha b$ and $r\perp b$. For any candidate $tb$,

$$x-tb=r+(\alpha-t)b.$$

The two terms are perpendicular, so Pythagoras gives

$$\lVert x-tb\rVert_2^2 =\lVert r\rVert_2^2+(\alpha-t)^2\lVert b\rVert_2^2.$$

The extra term is nonnegative and vanishes only when $t=\alpha$ because $b\ne0$. Thus the projection has the smallest distance over the whole line, not merely over the candidates we happened to sample.

Question 5

In the worked example x = (3, 1) and b = (2, 1), compare the candidate points for t = 1 and t = 2 with the projection. What are their distances, and why is the apparent symmetry not valid here?

Show the step-by-step solution

The projection coordinate is $\alpha=1.4$, and its distance is $\sqrt5/5\approx0.447$. For $t=1$, the point is $(2,1)$ and the difference from $x$ is $(1,0)$, so the distance is $1$. For $t=2$, the point is $(4,2)$ and the difference is $(-1,-1)$, so the distance is $\sqrt2\approx1.414$. They are not equal because the observation is not positioned symmetrically around the line coordinate interval $[1,2]$ in the Euclidean metric. Recomputing the point and difference prevents a copied table value from hiding an arithmetic mistake.

Question 6

An implementation prints a residual dot product of −2.2 × 10⁻¹⁶. Should it reject the projection because the value is not literally zero?

Show the step-by-step solution

No. The mathematical result is zero, while binary floating-point arithmetic can leave a tiny round-off remainder. Choose a tolerance that fits the scale of the inputs and check the result with a tolerance-aware comparison, for example np.isclose(np.dot(residual, b), 0.0, rtol=0.0, atol=1e-12). The tolerance should not be copied blindly for very large or very small data; it is part of the numerical contract.

Question 7

What changes if the model family is the affine line c + tb instead of the line through the origin?

Show the step-by-step solution

The offset changes which points are candidates, so first translate the observation by $c$:

$$y=x-c.$$

Project $y$ onto the direction $b$ using

$$\alpha=\frac{y^{\mathsf T}b}{b^{\mathsf T}b}.$$

Then translate back:

$$p=c+\alpha b =c+\frac{(x-c)^{\mathsf T}b}{b^{\mathsf T}b}b.$$

Using $x$ instead of $x-c$ would solve the origin-line problem and can place the result outside the intended affine model family.

Question 8

An AI engineer says, "The residual is orthogonal to the model direction, so it contains no useful information." Which part is correct, and which part is too strong?

Show the step-by-step solution

The correct part is geometric: the residual has zero dot product with the selected direction, so the model cannot reduce it by changing only the scalar coefficient along that direction. The conclusion about usefulness is too strong. The residual can contain another feature, a systematic error, noise, or an entire direction that the one-dimensional model omitted. Orthogonality is relative to the chosen subspace and metric; it is not a claim that the data remainder is meaningless.