Functions Can Have Inner Products Too

An inner product can tell whether two vectors point in similar directions. But a signal, curve, or time series may have far more than three coordinates. How can an AI system compare the shapes of two functions without pretending that a curve is just one arbitrary list of numbers?

The same idea extends naturally: multiply the functions point by point, then add those products across a domain. For a continuous domain, the sum becomes an integral. The result is a function inner product that can measure overlap, construct orthogonal basis functions, and turn a shape into features a model can use.

By the end of this lesson, you will be able to interpret $\langle f,g\rangle=\int_a^b f(x)g(x)\,dx$, calculate a small example exactly, approximate it from samples with the correct spacing, and explain why a generic similarity score is not automatically an inner product.

Learning outcome: Interpret function similarity through an integral or discrete sum.

This lesson connects Lesson 041: Inner Products Measure Alignment, where the dot product measured overlap between finite vectors, with Lesson 002: Functions as Reusable Mathematical Machines, where a function was a rule that turns each input into an output. The official Mathematics for Machine Learning book places this generalization in Section 3.7. The worked functions, sampled chart, NumPy output, AI connection, and assessments here are original.

From a finite dot product to a function inner product

For two real vectors $u=(u_1,\ldots,u_n)$ and $v=(v_1,\ldots,v_n)$, the ordinary inner product is

$$\langle u,v\rangle=\sum_{i=1}^{n}u_i v_i.$$

Each coordinate contributes a product. A positive product means the two values have the same sign at that coordinate; a negative product means that they pull in opposite directions. The sum keeps the contributions that agree and cancels contributions that disagree.

A real-valued function $f$ on an interval $[a,b]$ can be viewed as a vector with one coordinate for every input position $x$. There are infinitely many such positions, so the coordinate sum becomes an integral.

The definition inherits the useful behavior of a vector inner product when the functions are in a suitable integrable function space: it is linear in each argument, symmetric for real functions, and positive when a nonzero function is compared with itself. In particular,

$$\langle f,f\rangle=\int_a^b f(x)^2\,dx\geq 0.$$

The corresponding function norm is $\lVert f\rVert=\sqrt{\langle f,f\rangle}$. It measures total squared magnitude over the interval, not the height at one chosen point.

A worked integral: crossing shapes can still overlap

Use the interval $[0,1]$ and two simple shapes:

$$f(x)=1-x,\qquad g(x)=x.$$

One curve falls while the other rises. They meet at $x=0.5$, but both are nonnegative on the interval, so their pointwise products are never negative. Their inner product is

The visual below shows the same calculation in a form that does not require reading the integral first. The blue solid curve is $f$, the amber dashed curve is $g$, and the teal bars show the sampled products $f(x_i)g(x_i)$. The exact numbers remain available in the table, so the conclusion does not depend on color, a screen-size drawing, or client-side code.

Static function-space visual

Two curves and the products that measure their overlap

The two nonnegative curves produce positive products between the endpoints. The table makes every sampled value explicit.

The sampled values used by the overlap chart.

xᵢ	f(xᵢ)	g(xᵢ)	f(xᵢ)g(xᵢ)	What the product contributes
0	1	0	0	Zero at this endpoint
0.25	0.75	0.25	0.1875	Positive overlap
0.5	0.5	0.5	0.25	Positive overlap
0.75	0.25	0.75	0.1875	Positive overlap
1	0	1	0	Zero at this endpoint

The bars are a visual aid for the pointwise products. The integral is the continuous definition; a sample-based estimate must also account for spacing and endpoint weights.

Samples turn the integral into a weighted dot product

An implementation usually sees samples rather than a symbolic formula. Let $x_0,\ldots,x_m$ be sample positions and let $y_i=f(x_i)g(x_i)$. If the positions are equally spaced by $\Delta x$, a simple sum has the form

$$\Delta x\sum_{i=0}^{m}y_i.$$

That expression is useful intuition, but it counts both endpoints as heavily as interior points. The composite trapezoidal rule gives the first and last sample half weight:

$$\int_a^b f(x)g(x)\,dx \approx \Delta x\left(\frac{y_0}{2}+y_1+\cdots+y_{m-1}+\frac{y_m}{2}\right).$$

For the five samples in the visual, $\Delta x=0.25$ and $y=(0,0.1875,0.25,0.1875,0)$. Therefore,

$$0.25\left(0+0.1875+0.25+0.1875+0\right)=0.15625.$$

The estimate is close to the exact value $1/6\approx0.1667$, but it is not the same number. More samples would normally reduce the approximation error for these smooth curves.

The NumPy code mirrors the distinction. numpy.dot computes the one-dimensional inner product, so it is useful for the raw sampled sum. numpy.trapezoid integrates sampled values with the composite trapezoidal rule and uses the explicit x positions.

import numpy as np

x = np.linspace(0.0, 1.0, num=5)
f = 1.0 - x
g = x
products = f * g
raw_sample_sum = np.dot(f, g)
trapezoid_estimate = np.trapezoid(products, x=x)
weighted_sum = np.dot(np.array([0.5, 1.0, 1.0, 1.0, 0.5]), products)
weighted_sum *= x[1] - x[0]

print(products)
print(raw_sample_sum)
print(trapezoid_estimate)
print(weighted_sum)

Verified output:

[0.     0.1875 0.25   0.1875 0.    ]
0.625
0.15625
0.15625

The raw dot product is $0.625$, not the integral. It has no spacing factor and does not apply endpoint weights. The last two values agree because the explicit weighted dot product is the same composite trapezoid calculation that np.trapezoid performs. With uneven sample positions, pass those positions as x; do not silently substitute one global dx.

Basis functions turn shapes into model features

Function inner products become especially practical when a model describes an input shape with a small collection of basis functions. Consider the two functions on $[0,1]$

$$\phi_0(x)=1,\qquad \phi_1(x)=2x-1.$$

The first is constant and the second changes linearly from $-1$ to $1$. Their overlap is

$$\begin{aligned} \langle\phi_0,\phi_1\rangle_{[0,1]} &=\int_0^1(2x-1)\,dx\\ &=\left[x^2-x\right]_0^1=0. \end{aligned}$$

They are orthogonal under this interval and weight. The same functions are not automatically orthogonal if the domain or weight changes. Their lengths are $\lVert\phi_0\rVert=1$ and $\lVert\phi_1\rVert=\sqrt{1/3}$, so the pair is orthogonal but not orthonormal. Multiplying $\phi_1$ by $\sqrt{3}$ would make its norm one.

Suppose a feature-expansion model approximates a signal $h$ as

$$h(x)\approx c_0\phi_0(x)+c_1\phi_1(x).$$

If the basis is orthogonal, the coefficient for one direction does not mix with the other in the inner product. For example, if $h(x)=0.6\phi_0(x)-0.2\phi_1(x)$, then

$$\langle h,\phi_0\rangle =0.6\langle\phi_0,\phi_0\rangle -0.2\langle\phi_1,\phi_0\rangle =0.6.$$

The second term disappears because the basis functions are orthogonal. In practice, an AI pipeline can evaluate $\phi_0(x_i),\phi_1(x_i),\ldots$ at each sample and feed those values to a classifier, regressor, or other downstream model. The inner products describe how much an observed shape overlaps each chosen feature direction. They make the feature design inspectable; they do not guarantee that the chosen basis is sufficient for the task.

Why the domain, spacing, and metric matter

The phrase “similar functions” is not a complete mathematical specification. The inner product depends on choices that a production feature pipeline must make explicit.

Same domain. The integral compares values at the same $x$ positions. If one function is defined on $[0,1]$ and another on $[0,2]$, the expression $\int f(x)g(x)\,dx$ needs a stated common interval or a resampling rule. It is not meaningful to pair unmatched inputs by accident.

Sample spacing. A plain np.dot(f_samples, g_samples) treats each sample as one equally important coordinate. If samples are clustered in one part of the domain, that sum gives the cluster more weight. Use the actual positions or document the discrete geometry you intend.

Weighting changes the geometry. A nonnegative weight $w(x)$ can emphasize a time window or a region where measurements are more reliable:

$$\langle f,g\rangle_w=\int_a^b w(x)f(x)g(x)\,dx.$$

Changing $w$ changes which shapes count as orthogonal and which features receive more influence. That can be a deliberate modeling choice, not a harmless implementation detail.

Similarity is not automatically an inner product. A score such as $\max_x f(x)g(x)$ may be useful for a particular matching task, but it is not linear in the functions in the way an inner product is. Likewise, an arbitrary rescaled or thresholded score may be symmetric without being positive definite. Call a quantity an inner product only after checking the domain, integrability, linearity, symmetry, and positive-definiteness requirements appropriate to the function space.

Function inner products turn a vague question—“Do these shapes look alike?”—into a reproducible calculation. The integral supplies the continuous definition; np.dot and np.trapezoid make the sampled version explicit; and basis functions let an AI system compare interpretable shape directions. The next step is to use these inner products for projections, where a function is replaced by its closest approximation inside a chosen subspace.

Check your understanding

Question 1

Write the function inner product on [a, b] and explain what the product f(x)g(x) contributes at one point.

Show the step-by-step solution

The definition is

$$\langle f,g\rangle_{[a,b]}=\int_a^b f(x)g(x)\,dx.$$

At one point $x$, the product is positive when the two function values have the same sign, negative when they have opposite signs, and zero when at least one value is zero. The integral adds those local contributions across the stated domain, so the interval is part of the answer.

Question 2

Compute the inner product of f(x) = 1 and g(x) = x on [0, 1].

Show the step-by-step solution

First multiply the functions: $f(x)g(x)=x$. Then integrate over the supplied interval:

$$\langle f,g\rangle=\int_0^1x\,dx =\left[\frac{x^2}{2}\right]_0^1 =\frac12.$$

The result is positive because both functions are nonnegative on $[0,1]$, and the value depends on that interval.

Question 3

For f(x) = 1 − x and g(x) = x on [0, 1], verify the exact value 1/6 and the normalized overlap 1/2.

Show the step-by-step solution

The cross-inner product is

$$\int_0^1(1-x)x\,dx=\int_0^1(x-x^2)\,dx=\frac12-\frac13=\frac16.$$

For the norms, compute

$$\langle f,f\rangle=\int_0^1(1-x)^2dx=\frac13, \qquad \langle g,g\rangle=\int_0^1x^2dx=\frac13.$$

Thus the normalized overlap is $\frac{1/6}{\sqrt{1/3}\sqrt{1/3}}=\frac12$. It is below one because the curves are not scalar multiples of one another.

Question 4

Using the five samples x = (0, 0.25, 0.5, 0.75, 1), why is the raw dot product 0.625 different from the trapezoidal estimate 0.15625?

Show the step-by-step solution

The sampled products are $y=(0,0.1875,0.25,0.1875,0)$. The raw dot product between the sampled function arrays is

$$\sum_i f(x_i)g(x_i)=0+0.1875+0.25+0.1875+0=0.625.$$

That is an unweighted sample sum. The trapezoidal estimate multiplies by the spacing $\Delta x=0.25$ and gives each endpoint half weight:

$$0.25\left(\frac02+0.1875+0.25+0.1875+\frac02\right)=0.15625.$$

The two calculations answer different numerical questions; the second is the spacing-aware approximation to the integral.

Question 5

What is the inner product of φ₀(x) = 1 and φ₁(x) = 2x − 1 on [0, 1], and what does the result mean?

Show the step-by-step solution

Multiply the functions and integrate:

$$\langle\phi_0,\phi_1\rangle=\int_0^1(2x-1)dx =\left[x^2-x\right]_0^1=0.$$

The zero says the constant and centered linear shapes are orthogonal under the uniform inner product on $[0,1]$. It does not say that every possible definition of function similarity would call them unrelated; a new domain or weight defines a new geometry.

Question 6

A pipeline samples one function densely near x = 0.8 and sparsely elsewhere. Why can a plain np.dot of the samples distort the intended continuous inner product, and what should the implementation do?

Show the step-by-step solution

np.dot gives one equal coordinate weight to each supplied pair. The dense cluster therefore contributes more terms simply because it has more samples, even if the continuous domain should give that region no extra weight. The implementation should use the actual sample positions with np.trapezoid, or construct an explicitly justified quadrature weight for each sample. If the discrete weighting is intentional, document that it is a different inner product rather than calling it the uniform continuous integral.

Question 7

Why is S(f, g) = maxₓ f(x)g(x) not automatically an inner product even if it is useful as a matching score?

Show the step-by-step solution

An inner product must satisfy more than a similarity-like interpretation. It must be linear in each argument, symmetric for real functions, and positive definite on the chosen function space. The maximum operation selects one point and generally does not distribute over addition, so in general $S(f_1+f_2,g)\neq S(f_1,g)+S(f_2,g)$. Therefore the score may be useful, but it does not inherit the algebraic guarantees of an inner product.

Question 8

A feature-expansion model uses h(x) ≈ c₀φ₀(x) + c₁φ₁(x) with the orthogonal functions from the lesson. Why does ⟨h, φ₀⟩ isolate c₀ in the example h = 0.6φ₀ − 0.2φ₁?

Show the step-by-step solution

Use linearity in the first argument:

$$\langle h,\phi_0\rangle =0.6\langle\phi_0,\phi_0\rangle -0.2\langle\phi_1,\phi_0\rangle.$$

The basis functions are orthogonal, so the second inner product is zero. The first function has norm one, so $\langle\phi_0,\phi_0\rangle=1$. The result is $0.6$. The coefficient is isolated because the basis geometry removes the cross-term; with a nonorthogonal basis, the second term would remain and the coefficients would be coupled.