How Does Big Tech See Themselves? Our Saviors.
From 1 + 1 to ChatGPT in eight steps: weighted sums, activations, layers, gradient descent, embeddings, attention, the transformer stack and next-word prediction, with the code for every step.
The machine they call salvation
Big Tech describes its models in the language of rescue. They will cure disease, end drudgery, and some day perfect us. I wanted to see what is actually under that promise, so I rebuilt it from the bottom: one addition at a time, from 1 + 1 to the thing that writes back when you type into ChatGPT.
The film below is 70 seconds. Every number in it comes from running the companion repo, our-saviors on GitLab: nine folders, NumPy only, a runnable demo and tests in each. Nothing on screen was typed in by hand.
The film
How Does Big Tech See Themselves? Our Saviors.
1. A weighted sum
Every neuron starts with addition. Multiply each input by a weight, add them up, add a bias. With inputs 0.8 and 0.3, weights 0.5 and minus 1, and a bias of 0.1, the neuron outputs 0.2. The repo computes it twice, once with NumPy and once with a plain loop, to prove there is nothing hidden in the library.
c01_neuron/neuron.py
def weighted_sum(x, w, b=0.0):
x, w = np.asarray(x, float), np.asarray(w, float)
if x.shape != w.shape:
raise ValueError(f"x and w must match, got {x.shape} vs {w.shape}")
return float(np.dot(w, x) + b)2. A curve that decides
A sum on its own is a straight line, and stacking straight lines only ever gives you another straight line. The sigmoid bends it into a value between 0 and 1, something that behaves like a decision: sigmoid(0.4) is 0.599. Modern networks mostly use ReLU, which keeps positives and zeroes the rest, but the job is the same.
3. Layers are a matrix multiply
Put many neurons side by side and their weights form a grid. One matrix multiply computes all of them at once: y = sigma(Wx + b). The tiny network in the repo (5 inputs, two hidden layers of 8, 3 outputs) has 147 parameters. GPT-3 has about 175 billion. Same idea, different size.
4. Loss and gradient descent
At first every weight is wrong. The loss measures how wrong (here, the mean squared error), the gradient says which direction makes it worse, and each step moves every weight a little the other way. In the repo, the loss falls from 8.13 after the first step to 0.0025, and the learned weights land on the true ones, 2 and minus 3, within 0.006.
c04_gradient_descent/gradient_descent.py
def grads(w, b, X, y):
err = X @ w + b - y
n = len(y)
return 2 * X.T @ err / n, 2 * err.mean()
def train(X, y, lr=0.1, steps=500):
w, b = np.zeros(X.shape[1]), 0.0
history = []
for _ in range(steps):
gw, gb = grads(w, b, X, y)
w, b = w - lr * gw, b - lr * gb
history.append(mse(X @ w + b, y))
return w, b, historySteps 1 to 4
5. Words become coordinates
A model cannot read, so every word becomes a list of numbers. In the repo, Arcane vocabulary gets six hand-built axes (topside, undercity, technology, magic, power, danger). A real PCA squeezes them to two dimensions and keeps 84% of the variance: Piltover, Hextech, progress and the council gather on one side, Zaun, the Undercity, shimmer and the chem-barons on the other. Meaning becomes distance.
6. Attention
Then every word asks every other word how much it matters. Attention compares each word's query with every word's key, scales by the square root of the dimension, and a softmax turns the scores into weights that sum to 1. On "hextech will save the undercity", "will" puts 0.64 of its attention on "hextech", and "save" puts 0.63 on "will". A causal mask stops any word from looking ahead.
c06_attention/attention.py
def softmax(z, axis=-1):
z = z - z.max(axis=axis, keepdims=True)
e = np.exp(z)
return e / e.sum(axis=axis, keepdims=True)
def attention(Q, K, V, causal=False):
d = Q.shape[-1]
scores = Q @ K.T / np.sqrt(d)
if causal:
scores = scores + np.triu(np.full(scores.shape, -1e9), k=1)
weights = softmax(scores)
return weights @ V, weights7. The same block, stacked
A transformer block is attention plus a small feed-forward network, each wrapped in a residual connection and a layer norm. In the repo one block at width 16 has 3,072 weights. GPT-3 stacks 96 of them at width 12,288, which is roughly 96 x 12 x 12,288 squared, about 174 billion parameters, trained across thousands of processors.
8. Only ever the next word
All of that produces one thing: a probability for each possible next word. The model samples one, appends it, and does it again. In the repo's bigram model, after "save" the two options are "piltover" and "us" at 0.5 each, and at temperature 0.7 the sample is "us".
Steps 5 to 8
Every number is real
It is math, not magic
It is a beautiful machine, and it is understandable at every step. What it is not is a saviour. A machine that predicts us is not one that understands us, and the part of the story nobody can compute is who decides what it is for. That is the ninth breakdown in the series: alignment, where people rank answers and the model learns their idea of good.
Clone the repo, run python -m pytest, and change anything. The challenge list in the README has a few places to start.
Run it
git clone https://gitlab.com/j.michaelduhart/our-saviors.git
cd our-saviors
pip install -r requirements.txt
python -m pytest
python c06_attention/attention.pyThis week: one breakdown per step, starting with the weighted sum, the third grade math under every AI.
More: LinkedIn · Instagram. Portfolio and case studies: designsbyduhart.org.
If any of this saved you an afternoon, Buy me a coffee.