Linear Algebra Is All You Need: The Math Behind Every AI Model

Introduction

Every AI paper you read, every model you train, every inference you run reduces to the same thing: linear algebra. Not calculus, not probability, not information theory. Those matter, but they are not the substrate. The substrate is vectors, matrices, and the transformations between them.

A neural network's forward pass is a sequence of matrix multiplications interleaved with nonlinearities. Backpropagation computes gradients through those same matrices. Attention, the mechanism that made transformers dominant, is a structured dot product. Embeddings are vectors. Loss landscapes are surfaces in high-dimensional vector spaces. The entire edifice of modern AI is built on operations that a second-year linear algebra student could, in principle, perform by hand.

This post is not a textbook review. It is a guided tour of the specific linear algebra concepts that actually run inside every AI model, from the simplest feedforward network to GPT-scale transformers. If you understand these operations, you understand what the machine is doing. If you do not, no amount of API calls will give you that understanding.

Vectors: The Language of Data

Before anything can be processed by a neural network, it must become a vector: an ordered list of numbers in $\mathbb{R}^n$. This is not a metaphor. It is a literal requirement. Every input to every AI model is a point in some high-dimensional vector space.

A word becomes a vector through an embedding table. The word “king” might map to a vector in $\mathbb{R}^{768}$: 768 floating-point numbers that encode the word's semantic identity. An image becomes a vector (or a tensor of vectors) by reading pixel intensities into arrays. An audio waveform becomes a vector by sampling amplitudes at discrete time steps.

The key insight is that once data is in vector form, the same mathematical operations apply regardless of modality. A matrix multiply does not care whether the input vector represents a word, a pixel patch, or an audio frame. This is why the same transformer architecture works for text, images, and audio. The architecture operates on vectors. The embedding layer is the only part that knows what the data originally was.

Matrix Multiplication: The Core Operation

If there is one operation that defines neural networks, it is matrix multiplication. A single layer of a neural network computes:

where $\mathbf{x} \in \mathbb{R}^n$ is the input vector, $W \in \mathbb{R}^{m \times n}$ is the weight matrix, and $\mathbf{b} \in \mathbb{R}^m$ is a bias vector. The output $\mathbf{y} \in \mathbb{R}^m$ is a new vector in a potentially different-dimensional space. This is a linear transformation: the matrix $W$ rotates, scales, and projects the input.

A feedforward neural network is just this operation applied repeatedly with nonlinear activation functions between layers:

Without the activation function $\sigma$, the entire stack of layers would collapse into a single matrix multiplication: $W_3 W_2 W_1 \mathbf{x}$ is still a linear transformation. The nonlinearity is what gives depth its power. But the skeleton of every forward pass is matrix multiplication.

Dimensions Tell the Story

Reading a neural network's architecture is really reading a chain of matrix dimensions. A transformer block with hidden dimension $d = 768$ and MLP intermediate size $4d = 3072$ has weight matrices $W_1 \in \mathbb{R}^{3072 \times 768}$ and $W_2 \in \mathbb{R}^{768 \times 3072}$. The first expands the representation into a higher-dimensional space where the nonlinearity can separate features. The second projects it back. Every matrix shape is a design decision about how information flows.

Dot Products and Similarity

The dot product of two vectors $\mathbf{a}$ and $\mathbf{b}$ is defined as:

where $\theta$ is the angle between the vectors. This single operation is the foundation of similarity in AI. When two embedding vectors have a high dot product, they point in roughly the same direction in the embedding space. When the dot product is near zero, they are orthogonal, meaning unrelated.

Cosine similarity normalizes the dot product by the magnitudes of both vectors:

This gives a value between $-1$ and $1$, independent of vector magnitude. It measures direction, not scale, which is exactly what you want when comparing semantic meaning.

Eigenvalues and Eigenvectors

An eigenvector of a matrix $A$ is a nonzero vector $\mathbf{v}$ that, when multiplied by $A$, only gets scaled, not rotated:

The scalar $\lambda$ is the eigenvalue. In plain terms: eigenvectors are the directions along which a linear transformation acts as simple scaling. Every other vector gets both scaled and rotated. Eigenvectors are the special directions where the matrix's action is maximally simple.

Why does this matter for AI? Because of Principal Component Analysis (PCA). Given a dataset of $n$ vectors in $\mathbb{R}^d$, the covariance matrix $C \in \mathbb{R}^{d \times d}$ captures how features vary together. The eigenvectors of $C$ are the principal components, the directions along which the data varies most.

The eigenvalues of $C$ tell you how much variance lies along each principal component. If the first 50 eigenvalues capture 95% of the total variance, you can project your data onto those 50 eigenvectors and discard the rest. You have reduced dimensionality from $d$ to 50 with only 5% information loss.

Singular Value Decomposition

The Singular Value Decomposition (SVD) generalizes eigendecomposition to any matrix, not just square ones. Every matrix $A \in \mathbb{R}^{m \times n}$ can be decomposed as:

where $U \in \mathbb{R}^{m \times m}$ and $V \in \mathbb{R}^{n \times n}$ are orthogonal matrices (rotations/reflections), and $\Sigma \in \mathbb{R}^{m \times n}$ is a diagonal matrix of singular values $\sigma_1 \geq \sigma_2 \geq \cdots \geq 0$. The decomposition says: every linear transformation is a rotation, followed by axis-aligned scaling, followed by another rotation.

Low-Rank Approximation

The power of SVD lies in truncation. If you keep only the top $k$ singular values and their corresponding columns of $U$ and $V$, you get the best rank-$k$ approximation of the original matrix:

This is the Eckart-Young theorem: no other rank-$k$ matrix is closer to $A$ in the Frobenius norm. This result is everywhere in AI:

The Transformer Is Pure Linear Algebra

The transformer architecture, the foundation of GPT, BERT, and every modern language model, is a carefully orchestrated sequence of linear algebra operations. The core mechanism is scaled dot-product attention, and understanding it requires no concept beyond what we have already covered.

Given an input sequence of $n$ token embeddings packed into a matrix $X \in \mathbb{R}^{n \times d}$, the attention mechanism first computes three matrices through linear projections:

where $W_Q, W_K, W_V \in \mathbb{R}^{d \times d_k}$ are learned weight matrices. $Q$ is the query (what each token is looking for), $K$ is the key (what each token advertises about itself), and $V$ is the value (the actual information each token carries).

Attention is then computed as:

Let me unpack this operation by operation:

Multi-Head Attention

In practice, attention is computed in parallel across $h$ heads, each with its own projection matrices $W_Q^i, W_K^i, W_V^i \in \mathbb{R}^{d \times d_k}$ where $d_k = d / h$. The outputs are concatenated and projected back:

Each head learns to attend to different types of relationships: syntactic, semantic, positional. The concatenation and final projection $W_O \in \mathbb{R}^{d \times d}$ combine these perspectives into a single representation. The entire operation is matrix multiplications and a softmax. Nothing more.

Why GPUs Exist for This

Matrix multiplication is embarrassingly parallel. To compute $C = AB$ where $A \in \mathbb{R}^{m \times k}$ and $B \in \mathbb{R}^{k \times n}$, each element $C_{ij}$ is an independent dot product:

All $m \times n$ elements of $C$ can be computed simultaneously. A CPU with 16 cores can compute 16 dot products at once. An NVIDIA H100 GPU has 16,896 CUDA cores and can execute thousands of dot products per clock cycle through SIMD (Single Instruction, Multiple Data) parallelism.

This is not a coincidence. GPUs were originally designed for rendering 3D graphics, which requires transforming millions of vertices through $4 \times 4$ transformation matrices every frame. The same hardware that rotates polygons in video games multiplies weight matrices in neural networks. When NVIDIA pivoted to AI compute, they were not building new capability. They were repurposing existing capability for a structurally identical workload.

Python Verification

Theory is convincing. Code is conclusive. Let us verify the core linear algebra operations that run inside a neural network using NumPy. First, a minimal forward pass through a two-layer network:

import numpy as np

# Seed for reproducibility

np.random.seed(42)

# Two-layer feedforward network

d_in, d_hidden, d_out = 4, 8, 3
W1 = np.random.randn(d_hidden, d_in) * 0.5
b1 = np.zeros(d_hidden)
W2 = np.random.randn(d_out, d_hidden) * 0.5
b2 = np.zeros(d_out)

def relu(x):
    return np.maximum(0, x)

def forward(x):
    h = relu(W1 @ x + b1)   # Linear transform + activation

    y = W2 @ h + b2          # Output projection

    return y

x = np.array([1.0, 0.5, -0.3, 0.8])
output = forward(x)
print(f"Input:  {x}")
print(f"Output: {output}")
print(f"Shape:  {x.shape} → {output.shape}")
# The entire forward pass is two matrix multiplications

Every line of the forward pass is a matrix-vector product. The @ operator in NumPy is matrix multiplication. The ReLU is applied elementwise between the two linear transformations. That is the entire computation.

Now, the attention mechanism. This is a complete, working implementation of scaled dot-product attention in 15 lines:

import numpy as np

def softmax(x, axis=-1):
    e = np.exp(x - x.max(axis=axis, keepdims=True))
    return e / e.sum(axis=axis, keepdims=True)

def attention(X, W_Q, W_K, W_V):
    """Scaled dot-product attention from scratch."""
    Q = X @ W_Q               # Queries:  (n, d_k)

    K = X @ W_K               # Keys:     (n, d_k)

    V = X @ W_V               # Values:   (n, d_v)

    d_k = Q.shape[-1]
    scores = Q @ K.T / np.sqrt(d_k)  # Similarity matrix: (n, n)

    weights = softmax(scores)          # Attention weights:  (n, n)

    return weights @ V                 # Weighted values:    (n, d_v)


# Simulate a 6-token sequence with d=8

n_tokens, d_model, d_k = 6, 8, 4
X = np.random.randn(n_tokens, d_model)
W_Q = np.random.randn(d_model, d_k) * 0.5
W_K = np.random.randn(d_model, d_k) * 0.5
W_V = np.random.randn(d_model, d_k) * 0.5

output = attention(X, W_Q, W_K, W_V)
print(f"Input shape:   {X.shape}")          # (6, 8)

print(f"Output shape:  {output.shape}")     # (6, 4)

print(f"Attention out:\n{output.round(3)}")
# 3 matrix multiplications + softmax = attention

Count the linear algebra operations: three matrix multiplications to compute $Q$, $K$, $V$; one matrix multiplication for the scores $QK^\top$; one matrix multiplication for the output $\text{weights} \times V$. Five matrix multiplications and a softmax. That is the attention mechanism. There is nothing else hiding underneath.

Conclusion

Every concept in this post, vectors, matrix multiplication, dot products, eigenvalues, SVD, attention, maps directly to operations running inside AI models right now. These are not background theory. They are the actual computation.

When a language model generates a sentence, it is multiplying matrices. When an image model recognizes a face, it is computing dot products. When a recommendation engine suggests a song, it is measuring cosine similarity in a vector space. When a fine-tuned model adapts to a new task through LoRA, it is exploiting low-rank matrix structure. The math is not an abstraction over the computation. The math is the computation.

This is why I find linear algebra more illuminating than any framework tutorial. PyTorch and TensorFlow are interfaces. The operations beneath them are the same ones Euler, Gauss, and Sylvester studied centuries ago. Understanding those operations, not just calling functions that execute them, is what separates someone who uses AI from someone who understands it.

If you want to go deeper, the path is clear: implement things from scratch. Write a forward pass by hand. Build attention with nothing but NumPy. Compute an SVD and reconstruct an image from truncated singular values. The linear algebra will stop being abstract the moment you see it produce real outputs. And once it does, every AI paper you read will make more sense, because you will recognize the operations underneath the notation.