Orthogonal Complements Capture What Is Left

A model may explain an observation with a few learned directions and still leave something over. How can it tell whether that remainder is a genuinely new direction, rather than a hidden copy of what the model already explained?

The orthogonal complement gives a precise answer. It collects every direction that has zero inner product with an entire subspace, not merely with one chosen vector. For a feature plane in three dimensions, the complement is the line of normal directions. That line gives a clean home for a residual component: it has no linear overlap with the plane, while it can still have nonzero size and useful information.

By the end of this lesson, you will be able to describe an orthogonal complement, check its dimension and basis, interpret a plane’s normal vector, and use an orthonormal basis in NumPy to separate an explained component from a residual.

Learning outcome: Describe the directions orthogonal to a subspace.

This lesson builds on Lesson 045: Orthogonality Means No Linear Overlap, which made zero inner product a geometric test, and Lesson 036: Kernel and Image Explain Loss and Reach, which connected subspaces to information a linear map can lose or reach. Section 3.6 of the official Mathematics for Machine Learning book is the curricular reference. The definitions, numbers, visual, implementation, AI connection, and assessments here are original.

A subspace has a perpendicular remainder

Let $V=\mathbb{R}^D$ be a $D$-dimensional Euclidean vector space. Let $U$ be an $M$-dimensional subspace of $V$. A subspace is a collection of vectors that is closed under addition and scalar multiplication; it contains the zero vector and all linear combinations of its basis directions.

Three facts make the complement useful:

• If $U$ has dimension $M$ inside a $D$-dimensional space, then $\dim(U^\perp)=D-M$. Each independent direction in $U$ contributes one orthogonality constraint.

• $U$ and $U^\perp$ meet only at the zero vector: $U\cap U^\perp=\{0\}$. A nonzero vector cannot be both inside a subspace and perpendicular to every vector in that same subspace, because it would have a nonzero inner product with itself.

• In the standard Euclidean geometry, every $x\in\mathbb{R}^D$ has one decomposition

  $$x=u+r,\qquad u\in U,\quad r\in U^\perp.$$

  The first term is the part described by the subspace. The second is what the subspace cannot explain through its own directions.

The dimension rule is a counting guide, not a shortcut around the definition. The complement of a plane in $\mathbb{R}^3$ is one-dimensional, but the complement of a line in $\mathbb{R}^3$ is a two-dimensional plane. “The orthogonal complement” is not always a single arrow.

A plane and its normal line

Consider the plane

$$U=\operatorname{span}\left\{ b_1=\begin{bmatrix}1\\0\\1\end{bmatrix}, b_2=\begin{bmatrix}0\\1\\1\end{bmatrix} \right\}\subset\mathbb{R}^3.$$

An arbitrary vector in this plane is

$$\alpha b_1+\beta b_2 =\begin{bmatrix}\alpha\\\beta\\\alpha+\beta\end{bmatrix}.$$

So the plane consists of vectors satisfying $x_3=x_1+x_2$. To find a normal, look for $n=(n_1,n_2,n_3)$ that is orthogonal to both basis vectors:

$$n^{\mathsf T}b_1=n_1+n_3=0, \qquad n^{\mathsf T}b_2=n_2+n_3=0.$$

Choosing $n_3=1$ gives $n_1=n_2=-1$, so one normal is

$$n=\begin{bmatrix}-1\\-1\\1\end{bmatrix}.$$

Every scalar multiple of $n$ is also orthogonal to both $b_1$ and $b_2$. Because $U$ has dimension $2$ and the ambient space has dimension $3$, the complement has dimension $1$; therefore

$$U^\perp=\operatorname{span}\left\{ \begin{bmatrix}-1\\-1\\1\end{bmatrix} \right\}.$$

The diagram uses a plane for $U$ and an arrow for its normal direction. Its table is a complete numerical alternative to the drawing: it records both orthogonality tests and the dimension conclusion without relying on color, perspective, or JavaScript.

Static geometry visual

A two-dimensional subspace and its one-dimensional complement

The dark arrows span U. The orange arrow is n = (−1, −1, 1), a normal direction whose dot product with both basis vectors is zero.

The visual’s exact vector and orthogonality checks.

Vector	Dot-product test	Interpretation
b₁ = (1, 0, 1)	nᵀb₁ = −1 + 1 = 0	A basis direction in U
b₂ = (0, 1, 1)	nᵀb₂ = −1 + 1 = 0	A basis direction in U
n = (−1, −1, 1)	nᵀn = 1 + 1 + 1 = 3	n spans U⊥
x = (2, 1, 4)	nᵀx = −2 − 1 + 4 = 1	x has a nonzero normal component

The plane contains every linear combination of b₁ and b₂. The complement is the whole normal line, including both directions and the zero vector—not just the single arrow drawn here.

The visual uses an oblique drawing of three-dimensional space, so the screen angle is only a representation of the geometry. The algebra carries the meaning: the normal is perpendicular to every vector in $U$, because every vector in $U$ is a linear combination of $b_1$ and $b_2$.

Worked example: split an observation into explained signal and residual

The normal $n$ is a useful direction, but it is not unit length. To read coordinates cleanly, first use the orthonormal directions from the previous lessons. Gram–Schmidt applied to $b_1,b_2$ gives the following orthonormal basis for $U$:

$$q_1=\frac{1}{\sqrt{2}}\begin{bmatrix}1\\0\\1\end{bmatrix}, \qquad q_2=\frac{1}{\sqrt{6}}\begin{bmatrix}-1\\2\\1\end{bmatrix}.$$

The unit normal is

$$q_3=\frac{1}{\sqrt{3}}\begin{bmatrix}-1\\-1\\1\end{bmatrix}.$$

These three directions are pairwise orthonormal, and $q_1,q_2$ span $U$ while $q_3$ spans $U^\perp$. Take the observation

$$x=\begin{bmatrix}2\\1\\4\end{bmatrix}.$$

Because the basis is orthonormal, each coefficient is one inner product.

This calculation turns “explained versus left over” into a geometric statement. The word explained means “reconstructed from the selected subspace,” not “all useful information.” The residual can contain another feature, noise, a nonlinear pattern, or a signal that a later model should inspect.

Why the decomposition is unique

The uniqueness follows from the intersection fact rather than from a lucky choice of coordinates. Suppose the same vector had two decompositions:

$$x=u_1+r_1=u_2+r_2, \qquad u_1,u_2\in U,\quad r_1,r_2\in U^\perp.$$

Rearrange:

$$u_1-u_2=r_2-r_1.$$

The left side lies in $U$ because a subspace is closed under subtraction. The right side lies in $U^\perp$ for the same reason. Therefore this common vector lies in $U\cap U^\perp$, which contains only $0$. So $u_1=u_2$ and $r_1=r_2$.

The same proof explains why the two pieces do not double-count a direction. A nonzero vector cannot be both an explained direction in $U$ and a residual direction in $U^\perp$.

NumPy keeps the dimensions visible

Store the orthonormal basis vectors as the columns of a matrix $Q\in\mathbb{R}^{3\times2}$. The observation $x$ has shape $(3,)$, the coordinate vector $Q^{\mathsf T}x$ has shape $(2,)$, and the reconstructed explained vector $Q(Q^{\mathsf T}x)$ returns to shape $(3,)$.

The code below uses numpy.asarray to make the inputs arrays, the matrix-multiplication operator documented by numpy.matmul for the matrix products, numpy.linalg.norm for the Euclidean residual size, and numpy.allclose for tolerance-aware checks. The explicit atol and

rtol=0 make the intended zero comparisons visible; NumPy warns that a default absolute tolerance may be inappropriate near zero.

import numpy as np


def split_against_orthonormal_basis(x, Q, *, atol=1e-12):
    x = np.asarray(x, dtype=float)
    Q = np.asarray(Q, dtype=float)
    if x.ndim != 1 or Q.ndim != 2 or Q.shape[0] != x.shape[0]:
        raise ValueError('x must be (D,) and Q must be (D, M)')

    gram = Q.T @ Q
    if not np.allclose(
        gram, np.eye(Q.shape[1]), atol=atol, rtol=0.0
    ):
        raise ValueError('Q columns must be orthonormal')

    coordinates = Q.T @ x
    explained = Q @ coordinates
    residual = x - explained
    return coordinates, explained, residual


Q = np.array([
    [1 / np.sqrt(2), -1 / np.sqrt(6)],
    [0.0,             2 / np.sqrt(6)],
    [1 / np.sqrt(2),  1 / np.sqrt(6)],
])
x = np.array([2.0, 1.0, 4.0])

coordinates, explained, residual = split_against_orthonormal_basis(x, Q)
normal = np.array([-1.0, -1.0, 1.0]) / np.sqrt(3)

print(np.round(Q.T @ Q, 6))
print(np.round(coordinates, 6))
print(np.round(explained, 6))
print(np.round(residual, 6))
print(np.round(np.linalg.norm(residual), 6))
print(np.allclose(Q.T @ residual, 0.0, atol=1e-12, rtol=0.0))
print(np.allclose(residual, (1 / np.sqrt(3)) * normal, atol=1e-12, rtol=0.0))

The verified output is:

[[1. 0.]
 [0. 1.]]
[4.242641 1.632993]
[2.333333 1.333333 3.666667]
[-0.333333 -0.333333  0.333333]
0.57735
True
True

The first matrix is the $2\times2$ Gram matrix of the columns. Its identity shape confirms unit length and pairwise orthogonality. The final two checks say that the residual is orthogonal to both columns of $Q$ and lies on the unit normal direction. The function does not search for a nearest point or prove a least-squares fit; it uses an already verified orthonormal basis to expose the two coordinate blocks. That keeps this lesson focused on the complement. A later projection lesson can explain the nearest-point interpretation.

A residual is separated, not erased

In an AI representation, let $U$ be a subspace of feature directions selected by a linear encoder, a principal subspace, or a hand-designed feature family. For an input $x$, the decomposition $x=u+r$ gives:

• $u\in U$: the component expressible by the selected directions;
• $r\in U^\perp$: a component invisible to every linear score that only looks along $U$;
• $\lVert r\rVert_2$: the size of what remains under this geometry.

This can improve interpretability. If a linear readout is built only from the columns of $Q$, its score cannot obtain a contribution from $r$ through an inner product with those columns, because $Q^{\mathsf T}r=0$. That is a claim about this subspace, this inner product, and this representation—not a claim that the residual is useless. A nonlinear layer, a different metric, or a different dataset can reveal relationships that the current orthogonal split does not capture.

There are two boundary cases worth keeping close:

1. If $U=\mathbb{R}^D$, then $M=D$ and $U^\perp=\{0\}$. Everything is explained by the full space, so the only residual is zero.
2. If $U=\{0\}$, then $M=0$ and $U^\perp=\mathbb{R}^D$. Nothing is explained by the zero subspace, so every vector is residual.

The dimension rule also assumes the standard dot product. If a system defines a weighted inner product, “orthogonal” changes with that metric; the ordinary dot-product normal above is not automatically the correct residual direction.

The next lesson extends the same split into the geometry of the nearest point on a subspace. For now, the durable test is simpler: name the ambient space, write a basis for $U$, check zero inner products against that basis, and treat the complement as a subspace of all allowed residual directions.

Check your understanding

Question 1

Let U be an M-dimensional subspace of R^D. What is U⊥, and what dimension should it have?

Show the step-by-step solution

By definition, $U^\perp$ contains every vector $z\in\mathbb{R}^D$ whose inner product with every $u\in U$ is zero. It is enough to impose the test on an $M$-vector basis of $U$, because inner products distribute across linear combinations. Those independent orthogonality conditions leave $D-M$ independent directions, so $\dim(U^\perp)=D-M$ under the usual Euclidean geometry.

Question 2

For U = span{(1, 0, 1), (0, 1, 1)}, test n = (−1, −1, 1) and state U⊥.

Show the step-by-step solution

Test the first basis vector: $n^\mathsf{T}(1,0,1)=-1+1=0$. Test the second: $n^\mathsf{T}(0,1,1)=-1+1=0$. Because every vector in $U$ is a linear combination of those two basis vectors, $n$ is orthogonal to all of $U$. The two basis vectors are independent, so $\dim(U)=2$ in $\mathbb{R}^3$ and $\dim(U^\perp)=1$. Therefore $U^\perp=\operatorname{span}\{(-1,-1,1)\}$.

Question 3

Using the same plane, decide whether y = (2, 1, 3) has a residual component in U⊥. Show the test.

Show the step-by-step solution

The plane satisfies $x_3=x_1+x_2$. For $y$, $3=2+1$, so it already lies in $U$. The normal test says the same thing: $n^\mathsf{T} y=(-1)(2)+(-1)(1)+1(3)=-2-1+3=0$. Since $U^\perp$ is the normal line and the normal coordinate is zero, the residual component is the zero vector.

Question 4

For x = (2, 1, 4), use the worked decomposition to verify that the residual is orthogonal to U but is not zero.

Show the step-by-step solution

The decomposition gives $r=(-1/3,-1/3,1/3)$. It is not zero because its Euclidean norm is $\sqrt{1/9+1/9+1/9}=1/\sqrt{3}>0$. Check both basis directions: $b_1^\mathsf{T}r=1(-1/3)+0+1(1/3)=0$ and $b_2^\mathsf{T}r=0+1(-1/3)+1(1/3)=0$. Therefore the residual is a genuine nonzero direction in $U^\perp$, not an absence of information.

Question 5

Why is checking a candidate vector against only one vector in U insufficient when U is a plane?

Show the step-by-step solution

Orthogonality to a single vector removes only one linear overlap. A vector can be perpendicular to $b_1$ while still having a nonzero dot product with $b_2$, so it would not be perpendicular to every combination in the plane. For the candidate to belong to $U^\perp$, it must pass the zero-dot-product test for every basis direction of $U$. Passing those basis tests then covers every linear combination by distributivity.

Question 6

In the NumPy code, Q has shape (D, M). What are the shapes of Q.T @ x, Q @ (Q.T @ x), and x − Q @ (Q.T @ x)?

Show the step-by-step solution

$Q$ has shape $(D,M)$ and $x$ has shape $(D,)$. Its transpose has shape $(M,D)$, so $Q^\mathsf{T}x$ has shape $(M,)$: one coefficient per basis direction. Multiplying $Q$ by that coefficient vector returns shape $(D,)$, which is the explained component. Subtracting two vectors of shape $(D,)$ leaves a $(D,)$ residual in the original ambient space.

Question 7

What does the allclose check on Q.T @ Q verify, and why is an exact == comparison a poor floating-point test?

Show the step-by-step solution

For column-stored basis vectors, $Q^\mathsf{T}Q$ is their Gram matrix. An identity result means each column has unit norm and distinct columns have zero inner product, so the columns are orthonormal. Floating-point square roots and products can leave tiny values such as $2.2\times10^{-16}$ where exact arithmetic gives zero. np.allclose compares within explicit tolerances; the tolerance still needs to be chosen for the scale and precision of the data rather than accepted blindly.

Question 8

A model labels every residual in U⊥ as useless. Which part of that conclusion follows from the mathematics, and which part overreaches?

Show the step-by-step solution

The valid part is narrow: a residual $r\in U^\perp$ has zero inner product with every direction in $U$, so a linear readout that only scores those directions cannot recover a contribution from $r$ by that same inner product. The claim that $r$ is useless overreaches. It can have nonzero norm, encode another linear direction, carry nonlinear structure, or become useful to a later layer or a different metric.