Orthonormal Bases Simplify Coordinates

An AI representation can use a coordinate system that is easier to read than the raw feature axes. The most convenient case is an orthonormal basis: its directions are perpendicular, and each has length 1. In that basis, the amount of a signal along one direction is just one inner product. There is no small linear system to solve and no correction for a stretched axis.

By the end of this lesson, you will be able to verify that a basis is orthonormal and compute a vector’s coordinates with inner products. You will also know why orthogonal feature directions can reduce linear overlap without proving that the underlying random variables are independent.

Learning outcome: Verify orthonormality and compute coordinates with inner products.

This lesson combines Lesson 031: Bases Give Compact Coordinate Systems, which established unique coordinates in a basis, with Lesson 045: Orthogonality Means No Linear Overlap, which established the zero-inner-product test. The official Mathematics for Machine Learning book maps this idea to Section 3.5. The explanations, numerical example, code, visual, AI connection, and assessments here are original.

Perpendicular unit vectors make a readable frame

Let $u_1,\ldots,u_n$ be vectors in $\mathbb{R}^n$. A basis is orthonormal when every vector has unit Euclidean length and every distinct pair is orthogonal. The two requirements can be written in one rule:

The word basis still matters. A pair of perpendicular unit vectors in a plane reaches every point in that plane, so it is a basis for that plane. In a higher-dimensional space, the list must contain enough such directions to cover the space being represented. Orthonormality makes a basis especially easy to use; it does not change the dimension or create a new vector space.

Suppose a signal $v$ is written in this basis as

$$v=c_1u_1+c_2u_2+\cdots+c_nu_n.$$

Take the inner product with one basis vector, say $u_j$:

$$\begin{aligned} \langle u_j,v\rangle &=\left\langle u_j,\sum_{i=1}^{n}c_i u_i\right\rangle\\ &=\sum_{i=1}^{n}c_i\langle u_j,u_i\rangle\\ &=c_j. \end{aligned}$$

Every term disappears except the one with $i=j$: orthogonality makes the other inner products zero, and unit length makes the surviving inner product one. Therefore the coordinates are read directly as

$$\boxed{c_j=\langle u_j,v\rangle=u_j^{\mathsf T}v.}$$

For the full square basis matrix, this rule becomes two matching operations:

$$\boldsymbol{c}=U^{\mathsf T}v, \qquad v=U\boldsymbol{c}.$$

The transpose is the coordinate reader. The original matrix is the reconstructor.

A worked coordinate reading

Use the following two candidate directions:

$$u_1=\begin{bmatrix}3/5\\4/5\end{bmatrix}, \qquad u_2=\begin{bmatrix}-4/5\\3/5\end{bmatrix}.$$

They are rotated relative to the standard horizontal and vertical axes, but the fractions make the checks exact.

The coordinate reader below uses the same fixed basis and three small signals. Changing the signal changes the endpoint and the two inner products; the basis arrows stay perpendicular and unit length. The table is rendered with the visual, so the calculation remains available before JavaScript runs and to a reader who does not use the plot.

Read coordinates in an orthonormal basis

Fixed basis: u₁ = (3/5, 4/5), u₂ = (−4/5, 3/5)

Choose a signalSignal ASignal BSignal C

The table below remains the complete numerical alternative to the coordinate diagram.

Current reading: Signal A

Vector in standard coordinates

(2, 1)

Inner-product coordinates

(2, −1)

Reconstruction

(2, 1)

Every available signal and its coordinates in the fixed basis.

Signal	Vector	⟨v, u₁⟩	⟨v, u₂⟩
Signal A	(2, 1)	2	−1
Signal B	(1, 2)	2.2	0.4
Signal C	(−1, 1)	0.2	1.4

The coefficients are computed by inner products because the displayed basis is orthonormal.

NumPy can verify and read the basis

For one-dimensional arrays, NumPy defines numpy.inner as the ordinary inner product. numpy.linalg.norm computes a vector norm; with its default order on a one-dimensional array, that is the Euclidean length used here. The following code keeps the basis vectors as columns of $U$, so U.T @ v performs the two inner products in one operation.

import numpy as np

U = np.array([
    [3 / 5, -4 / 5],
    [4 / 5,  3 / 5],
], dtype=float)  # u1 and u2 are columns
v = np.array([2.0, 1.0])

gram = U.T @ U
coordinates = U.T @ v
reconstructed = U @ coordinates

print(np.round(gram, 3))
print(coordinates)
print(reconstructed)
print(np.allclose(gram, np.eye(2)))
print(np.allclose(reconstructed, v))

The output is:

[[1. 0.]
 [0. 1.]]
[ 2. -1.]
[2. 1.]
True
True

U.T @ U checks orthonormality, U.T @ v reads the coordinates, and U @ coordinates reconstructs the signal. The allclose calls are appropriate for floating-point calculations: a decimal implementation may produce a tiny value such as 2.220446049250313e-16 where the exact fraction is zero. They do not turn an arbitrary matrix into an orthonormal basis; they only compare the computed result with a tolerance.

An orientation mistake is easy to make. If basis vectors are stored as rows, the multiplication order changes. Decide whether rows or columns carry the directions, write down the intended shape, and check reconstruction rather than assuming that a multiplication that happens to fit is meaningful.

Why this helps an AI representation

Suppose an encoder represents a centered signal $v$ in two learned directions. If the directions form an orthonormal basis, the representation can store $c=U^{\mathsf T}v$. Each coordinate reports the signal’s signed amount along one direction, and the squared length separates cleanly:

$$\lVert v\rVert_2^2 =\lVert c_1u_1+c_2u_2\rVert_2^2 =c_1^2+c_2^2.$$

The cross-term disappears because $u_1^{\mathsf T}u_2=0$. A downstream linear readout can therefore reason about one coordinate without accidentally adding a component of the other through the geometry of the basis.

This is the geometric meaning of decorrelated axes: the chosen directions have no linear overlap under the specified inner product. It is useful, but it is a narrower statement than “the features are unrelated.”

Principal component analysis is a familiar implementation setting. The current scikit-learn PCA documentation describes PCA as projecting centered data onto orthogonal components. Its whiten=True option additionally scales the transformed components to have unit component-wise variance; ordinary orthogonality does not imply that all components have the same variance. PCA can therefore provide an orthogonal coordinate frame, while whitening is a separate scale choice.

Three boundaries that prevent overclaiming

A non-orthonormal basis still works, but the shortcut does not

Take $b_1=(1,0)^{\mathsf T}$ and $b_2=(1,1)^{\mathsf T}$. They form a basis for $\mathbb{R}^2$: they are independent and span the plane. But $b_2$ has length $\sqrt{2}$, and the two vectors are not orthogonal. For $v=(2,3)^{\mathsf T}$, the inner products are

$$\langle b_1,v\rangle=2, \qquad \langle b_2,v\rangle=5,$$

which are not its coordinates. Solve the actual combination instead:

$$c_1b_1+c_2b_2 =\begin{bmatrix}c_1+c_2\\c_2\end{bmatrix} =\begin{bmatrix}2\\3\end{bmatrix}.$$

The second row gives $c_2=3$, and then the first gives $c_1=-1$. The coordinates are $(-1,3)$, not $(2,5)$. If a basis is orthogonal but its vectors are not unit length, a partial shortcut survives:

$$c_j=\frac{\langle b_j,v\rangle}{\lVert b_j\rVert_2^2}.$$

For a fully non-orthogonal basis, the cross-terms remain and the coordinate problem is a linear system involving the basis’s Gram matrix.

The zero vector has no direction to normalize

The zero vector satisfies $0^{\mathsf T}x=0$ for every $x$, so the algebraic definition calls it orthogonal to every vector. But $\lVert0\rVert_2=0$ and $0/\lVert0\rVert_2$ is undefined. It cannot be a member of an orthonormal basis: it has no direction and it is linearly dependent with every list that contains it. A zero vector can be a signal whose coordinates are all zero; it cannot be one of the axes used to read those coordinates.

Orthogonality is not statistical independence

These words answer different questions:

Term	What is being compared?	Zero condition means	What it does not prove
Orthogonality	Fixed vectors in a chosen inner-product space	Their inner product is zero	No nonlinear relationship in data
Decorrelation	Centered random variables or dataset columns	Their covariance is zero under the stated sample convention	Statistical independence
Independence	A joint probability distribution	The joint law factors into marginal laws	It is not established by one dot product

Here is a small counterexample. Let

$$x=(-1,0,1), \qquad y=(1,-2,1)=3x^2-2$$

where the last equality is evaluated coordinate by coordinate. Both columns are centered, and

$$x^{\mathsf T}y=(-1)(1)+0(-2)+1(1)=0.$$

Their empirical covariance is therefore zero, but $y$ is completely determined by $x$ through a nonlinear rule. A linear overlap test missed that dependency; it did not establish independence. In an AI pipeline, call the result “orthogonal” or “uncorrelated on this centered sample” unless a separate probabilistic argument supports the stronger claim.

Check your understanding

Question 1

Do u₁=(0, 1) and u₂=(−1, 0) form an orthonormal basis for ℝ²? Verify both requirements and explain the conclusion.

Show the step-by-step solution

Compute the two lengths:

$$\lVert u_1\rVert_2=\sqrt{0^2+1^2}=1, \qquad \lVert u_2\rVert_2=\sqrt{(-1)^2+0^2}=1.$$

Now compute the cross-inner-product:

$$u_1^{\mathsf T}u_2=0(-1)+1(0)=0.$$

The vectors are unit length and perpendicular. They are also two independent directions in $\mathbb{R}^2$, so they form an orthonormal basis.

Question 2

Using u₁=(3/5, 4/5) and u₂=(−4/5, 3/5), compute the coordinates of v=(1, 2) with inner products.

Show the step-by-step solution

Because the basis is orthonormal, compute one inner product for each coordinate:

$$\begin{aligned} c_1&=u_1^{\mathsf T}v=\frac35(1)+\frac45(2)=\frac{11}{5},\\ c_2&=u_2^{\mathsf T}v=-\frac45(1)+\frac35(2)=\frac25. \end{aligned}$$

So $[v]_U=(11/5,2/5)^{\mathsf T}$. Check the first component of the reconstruction:

$$\frac{11}{5}\frac35+\frac25\left(-\frac45\right) =\frac{33-8}{25}=1.$$

The second component is

$$\frac{11}{5}\frac45+\frac25\frac35 =\frac{44+6}{25}=2,$$

so the coordinates reconstruct $(1,2)$.

Question 3

What standard-coordinate vector does the coordinate pair (3, −2) represent in the fixed basis from the lesson? Show the combination before simplifying.

Show the step-by-step solution

Start with the definition of basis coordinates:

$$v=3u_1-2u_2 =3\begin{bmatrix}3/5\\4/5\end{bmatrix} -2\begin{bmatrix}-4/5\\3/5\end{bmatrix}.$$

Combine corresponding components:

$$v=\begin{bmatrix}9/5+8/5\\12/5-6/5\end{bmatrix} =\begin{bmatrix}17/5\\6/5\end{bmatrix}.$$

The coordinate pair is not itself the standard-coordinate vector; it is the two coefficients that must be applied to the two basis directions.

Question 4

Why does cⱼ=⟨uⱼ,v⟩ isolate one coordinate when v=Σᵢ cᵢuᵢ? Give the two properties that remove the other terms.

Show the step-by-step solution

Take the inner product with $u_j$ and distribute it over the sum:

$$\langle u_j,v\rangle =\left\langle u_j,\sum_i c_i u_i\right\rangle =\sum_i c_i\langle u_j,u_i\rangle.$$

For every $i\ne j$, orthogonality gives $\langle u_j,u_i\rangle=0$, so those terms vanish. For $i=j$, unit length gives $\langle u_j,u_j\rangle=1$, leaving $c_j\cdot1=c_j$. Both properties are needed: perpendicular but stretched vectors leave a length-squared factor, while unit but non-perpendicular vectors leave cross-terms.

Question 5

For b₁=(1, 0), b₂=(1, 1), and v=(2, 3), explain why the inner products are not coordinates and solve for the actual coordinates.

Show the step-by-step solution

First check the shortcut’s inputs. The second vector has length $\sqrt2$ and

$$b_1^{\mathsf T}b_2=1(1)+0(1)=1\ne0,$$

so the basis is neither orthogonal nor orthonormal. The inner products are $b_1^{\mathsf T}v=2$ and $b_2^{\mathsf T}v=5$, but these numbers do not have the shortcut’s interpretation.

Solve the defining combination:

$$c_1b_1+c_2b_2 =\begin{bmatrix}c_1+c_2\\c_2\end{bmatrix} =\begin{bmatrix}2\\3\end{bmatrix}.$$

The second row gives $c_2=3$; substituting into the first gives $c_1+3=2$, hence $c_1=-1$. The actual coordinates are $(-1,3)$.

Question 6

A colleague says the zero vector is a valid orthonormal direction because its dot product with every vector is zero. What is wrong with that reasoning?

Show the step-by-step solution

The dot-product statement is algebraically true: $0^{\mathsf T}x=0$. But an orthonormal vector must also have norm 1, while

$$\lVert0\rVert_2=0.$$

There is no direction to normalize, since dividing by its norm would require division by zero. The zero vector is therefore excluded from an orthonormal basis even though it is orthogonal in the algebraic zero-inner-product sense.

Question 7

The centered samples x=(−1, 0, 1) and y=(1, −2, 1) have zero dot product. Does that prove x and y are statistically independent? Use the relation between them to decide.

Show the step-by-step solution

Their dot product is

$$(-1)(1)+0(-2)+1(1)=0,$$

so their empirical covariance is zero under the centered-sample convention. However, every value of $y$ is determined by $x$ through $y=3x^2-2$. That is a deterministic nonlinear relationship, so the variables are not independent. The zero dot product rules out measured linear co-movement in this sample; it does not rule out nonlinear dependency.

Question 8

In the NumPy code, U stores basis directions as columns. A bug replaces coordinates = U.T @ v with coordinates = U @ v. What check should catch the mistake, and why is U.T the correct reader?

Show the step-by-step solution

The reconstruction check should compare U @ coordinates with v using np.allclose. The intended coordinate equation is

$$c=U^{\mathsf T}v,$$

because its $j$th entry is the dot product of column $u_j$ with $v$. Then

$$Uc=U(U^{\mathsf T}v)=v$$

for a full orthonormal basis. Replacing $U^{\mathsf T}$ with $U$ computes a different linear combination of the rows and columns; it may still produce a vector with the right shape, but the reconstruction comparison exposes that it does not contain the intended inner-product coordinates.