From vectors to embeddings: the geometry behind similarity

An embedding represents an object—a word, image, user, or document—as a list of numbers. That list is a vector:

$$\mathbf{x} = [x_1, x_2, \ldots, x_n].$$

The individual coordinates rarely have a simple human label. What matters is the geometry created by many vectors together.

Start with direction

Suppose two vectors point in nearly the same direction. Their coordinates may have different magnitudes, but they still express a similar pattern. Cosine similarity measures that directional agreement:

$$\operatorname{cos}(\mathbf{x}, \mathbf{y}) = \frac{\mathbf{x} \cdot \mathbf{y}} {\lVert \mathbf{x} \rVert_2\,\lVert \mathbf{y} \rVert_2}.$$

The numerator is the dot product. The denominator removes the effect of length.

A small worked example

Let $\mathbf{x} = [1, 2]$ and $\mathbf{y} = [2, 4]$. Because $\mathbf{y} = 2\mathbf{x}$, the vectors point in exactly the same direction and their cosine similarity is $1$.

function dot(left, right) {
  return left.reduce((sum, value, index) => sum + value * right[index], 0);
}

The code mirrors the mathematical definition $\mathbf{x}\cdot\mathbf{y}=\sum_i x_i y_i$.

Why embeddings are useful

Training arranges vectors so that useful relationships become geometric relationships. A search system can therefore retrieve nearby documents; a recommendation system can compare users with items; and a classifier can draw boundaries between regions.

The important idea is simple: an embedding is useful when distance or direction preserves a relationship we care about.