How AI Really Works: Tokens, Transformers & the Hardware That Unlocked It

I grew up on the seam between two worlds. The analog ’80s, where a TV had a dial and you fixed things by smacking them. Then the tech ’90s, where the soundtrack of the internet was a modem screaming at a phone line until the phone line gave up and let it in. I’ve spent more than twenty years since then building networks, carrying voice traffic, and keeping a lot of blinking boxes alive. I’m a technologist at heart, so when modern AI hit the scene several years back, of course I went 10 toes down.

Here’s my take, up front, so you can argue with me in the comments.

The AI boom is not mostly a story about a brilliant new idea. A lot of the core ideas are decades old. Some of them are older than me. What changed is that the hardware finally caught up with the math. We spent fifty years with the right recipe and a kitchen the size of a phone booth.

To make that case properly, I need to walk through how this stuff actually works. Not the marketing version. The real version, with tokens and tensors and matrix multiplication. I’ll keep it plainspoken, and I’ll use network analogies where they fit, because that’s how my brain is wired.

Toward the end I’ll also take on the Skynet question, because I know it’s on a lot of people’s minds.

Grab a coffee. This one is long.

AI, ML, deep learning: sorting out the acronyms

These words get thrown around like they mean the same thing. They don’t.

Artificial intelligence is the big umbrella. Any technique that gets a machine to do something we’d call “smart” counts. A chess program built from hand-written rules is AI. So is a spam filter. So is the thing writing your kid’s book report.

Machine learning is a subset of AI. Instead of a human writing the rules, you show the system examples and let it figure out the rules on its own. You don’t tell a spam filter “the word lottery is bad.” You feed it a million emails labeled spam or not-spam and let it learn which patterns matter.

Deep learning is a subset of machine learning. It uses neural networks with many layers stacked on top of each other (that’s the “deep”). Each layer learns to transform its input into something a little more useful for the layer above it.

NLP, natural language processing, is the field that cares about human language specifically. Translation, speech recognition, summarizing, chatbots. NLP is older than deep learning and used plenty of other techniques for most of its history.

LLMs, large language models, are where those circles overlap today. They are deep learning models, trained with machine learning, doing NLP, and they sit under the AI umbrella. ChatGPT, Claude, Grok, Gemini, Llama. Same basic animal.

If you’re a network person, think of it like the OSI model in reverse. AI is the whole stack. LLMs are one very specific protocol running near the top.

The oldest trick in the book: guess the next symbol

At its core, an LLM does one thing. It looks at a sequence of text and predicts what comes next. One chunk at a time. That’s it. Everything else, the essays, the code, the poems about your cat, falls out of doing that one thing extremely well.

And that idea is from 1948.

Claude Shannon, the father of information theory, published “A Mathematical Theory of Communication” that year. If you’ve ever done a link budget or worried about signal-to-noise ratio, you’ve used his work, whether you knew it or not. In that same paper he played a game with English. He built text by choosing each letter based on the statistics of the letters before it. First with no context. Then with one previous letter. Then two. Then he did it with whole words.

The output got more English-looking as the context got longer. Gibberish turned into something that sounded almost like a sentence, even though it meant nothing. Those are what we now call n-gram models: predict the next item from the previous n-1 items. A few years later he went further and had people guess the next letter of real text to estimate how much information English actually carries per character.

Read that again. In 1948 and the early ’50s, the person who gave us the math behind every modem and every radio link was already doing next-token prediction, by hand, with a pencil and a book.

Shannon framed language as a signal you can predict. The better you predict it, the less surprise is left. Surprise is information. Everything that isn’t surprise is redundancy you could, in principle, compress away. Hold onto that. It’ll come back.

The talkers: ELIZA and A.L.I.C.E.

The next ancestors in this family tree didn’t learn anything at all. They’re still worth a stop.

ELIZA came out of MIT in 1966, written by Joseph Weizenbaum. Its most famous script, DOCTOR, played a Rogerian therapist. You typed “I’m worried about my job,” and it said “Why are you worried about your job?” It worked by matching keywords and reflecting your own words back with some pronouns flipped. No understanding. No learning. People got attached to it anyway, which bothered Weizenbaum quite a bit.

Jump to 1995. Richard Wallace started building A.L.I.C.E. (Artificial Linguistic Internet Computer Entity). It ran on AIML, the Artificial Intelligence Markup Language, which Wallace created for it. AIML is basically a giant pile of pattern-and-response rules written in XML. A pattern like WHAT IS YOUR NAME maps to a template response. Wildcards let one rule cover a lot of phrasing. Some rules redirect to other rules, so a well-built AIML bot can feel surprisingly fluid.

<category>
  <pattern>WHAT IS *</pattern>
  <template>I am not sure what <star/> is. Can you tell me?</template>
</category>

A.L.I.C.E. won the Loebner Prize, an annual Turing-test-style contest, in 2000, 2001 and 2004.

I want to be straight about this, because I see it get muddled online. A.L.I.C.E. was not a primitive LLM. It wasn’t trained on data in the machine learning sense. It had no neural network, no probabilities, no learned weights. It was a huge, carefully hand-built rulebook. Brilliant work, and a lot of it, but a rulebook.

That actually makes my point stronger. For decades, the best way to make a machine seem to talk was to have humans write tens of thousands of rules by hand, because the learning approach couldn’t be run at the scale it needed. The ambition was always there. The machines we had were the bottleneck.

Meanwhile, the statisticians were quietly winning

While the chatbot crowd wrote rules, the speech recognition and machine translation crowd went full Shannon. Through the ’90s and 2000s, statistical n-gram language models were the workhorse. Count how often word sequences appear in a big pile of text, apply some smoothing so you don’t assign zero probability to things you just haven’t seen yet, and use those counts to score which next word is most likely.

They worked. They powered real products. But they had a hard ceiling. An n-gram model only sees a few words back, typically three to five. And the number of possible word combinations explodes as n grows. With a 50,000-word vocabulary, the number of possible 5-word sequences is 50,000 to the fifth power, about 3 × 10²³. You will never see most of them, no matter how much text you count. That’s the curse of dimensionality, and it’s a wall.

Then, in 2003, Yoshua Bengio and colleagues published “A Neural Probabilistic Language Model.” It’s one of those papers that looks modest and turns out to be a foundation stone. Two ideas in it matter a lot:

First, represent every word as a learned vector of numbers, a point in space, instead of an arbitrary ID. Words used in similar ways end up close to each other. That’s an embedding.

Second, use a neural network to predict the next word from those vectors. Because similar words have similar vectors, the model can generalize. If it has seen “the cat sat on the mat,” it has a head start on “the dog sat on the rug,” even if it never saw that exact sentence.

That’s the core of a modern LLM, in 2003. The paper even talks about how expensive the thing was to train and spends real effort on speeding it up by spreading it over multiple CPUs. Keep that in mind.

Neural networks, gradient descent and backprop

Let’s open up the box.

A neural network is a big function with a lot of adjustable knobs. The knobs are called parameters (or weights). When someone says a model has 7 billion parameters, they mean 7 billion numbers, each one a knob that was tuned during training.

A single layer usually does something like this:

y = activation(W · x + b)

x is the input, a list of numbers. W is a matrix of weights. b is a bias vector. The activation is a simple non-linear function, something like “if negative, make it zero.” Without that non-linearity, stacking layers would collapse into one big linear function and you’d gain nothing. With it, a deep stack of layers can approximate shockingly complicated functions.

So how do you set billions of knobs? You start them at random and nudge them, over and over.

Feed in an example. Get a prediction.
Measure how wrong it is with a loss function. For language models that’s usually cross-entropy, which is a direct descendant of Shannon’s math. It measures how surprised the model was by the correct next token.
Work out, for every single parameter, which direction would make the loss a little smaller. That direction is the gradient.
Move every parameter a tiny step that way. That’s gradient descent.
Repeat. Billions of times.

Step 3 is the hard part. A model has billions of parameters, and you need the gradient for all of them on every step. The trick that makes it practical is backpropagation, popularized by Rumelhart, Hinton and Williams in their 1986 Nature paper. Backprop uses the chain rule from calculus to push the error backwards through the network, layer by layer, computing every gradient in roughly the same amount of work as a couple of forward passes.

I think of it like troubleshooting a bad circuit hop by hop. You know the far end is getting garbage. You walk back through each device and figure out how much each one contributed to the mess, and you fix a little at every hop instead of ripping out the whole path.

The training algorithm behind every modern model is from the year Top Gun came out.

In 1997, Sepp Hochreiter and Jürgen Schmidhuber added LSTMs (Long Short-Term Memory networks), a kind of recurrent network that reads text one token at a time and carries a memory along with it. LSTMs were a real step forward for sequences, and they dominated language and speech tasks into the mid-2010s.

So by the late ’90s we had: next-symbol prediction as a framing (1948), backprop (1986), and recurrent networks that could handle sequences (1997). Bengio’s word vectors showed up in 2003. A lot of the intellectual toolbox was sitting right there.

What we didn’t have was the horsepower.

Tokens: how text becomes something a model can chew on

Models don’t read letters or words. They read tokens.

A token is a chunk of text: sometimes a whole word, sometimes part of one, sometimes a single character or a piece of punctuation. Most modern LLMs build their vocabulary with some flavor of byte pair encoding (BPE), an old compression idea repurposed for language.

BPE works like this:

Start with the smallest units, individual bytes or characters.
Scan a huge pile of training text and find the pair of adjacent units that shows up most often.
Merge that pair into a new single unit and add it to the vocabulary.
Repeat until the vocabulary hits a target size, often somewhere from about 32,000 to a couple hundred thousand tokens.

Common words end up as single tokens. Rare words get split into pieces. A word like “networking” might come out as “network” + “ing.” Something weird like “NetworkNerd1337” will get chopped into several chunks.

"Signal and noise" -> ["Signal", " and", " noise"]   # roughly 3 tokens

A handy rule of thumb for English is that a token averages about three-quarters of a word, so 1,000 tokens is roughly 750 words. It varies a lot with language and content. Code and non-English text usually cost more tokens per word.

If you’ve worked with voice, tokenization is a bit like a codec. Before you can move a voice call over IP, you chop the audio into frames and encode them into a compact format. Tokenization chops text into units the model can move around efficiently. Pick a bad codec and quality suffers. Pick a bad tokenizer and the model spends its capacity on dumb stuff, like counting letters it can’t actually see. (This is the real reason models used to flub “how many r’s are in strawberry.” They see tokens, not letters.)

Each token gets an ID number, and that ID gets looked up in a big table to get its embedding.

Embeddings: meaning as a location

An embedding is a vector, a list of numbers, that represents a token. In a big model, each embedding might be a few thousand numbers long. GPT-3’s largest model used 12,288.

You can think of each embedding as a coordinate in a space with thousands of dimensions. Nobody designs those coordinates by hand. They start random and get learned during training, just like every other weight. Over time, tokens used in similar contexts drift toward each other. “Router” ends up near “switch.” “Ham” ends up near “radio” in one direction and near “sandwich” in another. Directions in that space start to carry meaning.

Humans can’t picture 4,000 dimensions. I certainly can’t. But the math doesn’t care. Distance and direction work the same way they do in 2D or 3D, there’s just a lot more room.

Tensors: the shape of everything

Here’s a word that gets a lot of hype: tensor. In machine learning, a tensor is just a multi-dimensional array of numbers. Nothing mystical.

A single number is a scalar, a rank-0 tensor.
A list of numbers is a vector, rank 1. Shape: [4096].
A grid of numbers is a matrix, rank 2. Shape: [4096, 4096].
Stack matrices and you get rank 3 and up.

(Physicists and mathematicians mean something more specific by “tensor,” a geometric object that transforms in particular ways when you change coordinates. ML borrowed the word and mostly uses it to mean “n-dimensional array.” Fine by me.)

Inside a language model, the data flowing through is usually a rank-3 tensor:

[batch, sequence_length, d_model]
e.g. [8, 2048, 4096]

That’s 8 separate text sequences, each 2,048 tokens long, with each token represented by 4,096 numbers. About 67 million numbers in that one tensor, and that’s just the activations moving through one point in the model.

The workhorse operation on all of it is matrix multiplication. Multiply an [m × k] matrix by a [k × n] matrix and you get an [m × n] result. Each output number is a dot product: multiply k pairs of numbers and add them up.

C[i][j] = sum over p of A[i][p] * B[p][j]
cost  ~= 2 * m * k * n  floating-point operations (FLOPs)

So multiplying two 4,096 × 4,096 matrices takes about 2 × 4,096³, roughly 137 billion floating-point operations. Once. A big model does mountains of these for every token.

Here’s the beautiful thing about matrix multiplication: every one of those output numbers can be computed independently. Nothing waits on anything else. It is what computer folks call “embarrassingly parallel.” Remember that sentence. It’s the whole hardware story in miniature.

Transformers and self-attention

In 2017, a team at Google published “Attention Is All You Need” (Vaswani et al.). It introduced the transformer, and it’s the T in GPT.

Before transformers, the go-to for language was recurrent networks like LSTMs. They read text one token at a time, left to right, carrying a memory forward. That has two problems. First, information from far back tends to fade by the time you need it. Second, and this is the one I care about for this post, it’s sequential. You can’t process token 500 until you’ve processed token 499. That’s a single-lane road, and it leaves parallel hardware sitting idle.

The transformer threw out the recurrence. Every token looks at every other token directly, all at once. That mechanism is self-attention.

Here’s how it works. For each token, the model takes its current vector and multiplies it by three learned weight matrices to make three new vectors:

Query (Q): what this token is looking for.
Key (K): what this token has to offer, as a label.
Value (V): the actual information it will hand over if picked.

Then:

Attention(Q, K, V) = softmax( Q · Kᵀ / √d_k ) · V

In plain English:

Compare every token’s query against every token’s key with a dot product. A big score means “this one is relevant to me.”
Divide by the square root of the key size so the numbers don’t blow up.
Softmax turns each row of scores into weights that add up to 1.
Each token’s new representation is a weighted blend of everyone’s values.

In a language model there’s also a mask so a token can only look backwards, at tokens before it. Otherwise it could peek at the answer it’s supposed to predict.

Take “The technician replaced the router because it was failing.” When the model processes “it,” attention lets it put heavy weight on “router” and little on “technician.” It learned that from data. Nobody wrote a rule.

Transformers run many attention computations side by side, called heads, each free to learn a different kind of relationship. Then a feed-forward block (more big matrix multiplications) processes each token. That attention-plus-feed-forward combo is one layer, and big models stack dozens to over a hundred of them.

My network brain sees it this way. An RNN is a daisy chain. Every packet has to pass through every hop in order. Self-attention is a full mesh. Every node talks to every other node directly in one step. Full mesh is great for latency and terrible for scale, which brings us to context windows.

Context windows and the quadratic problem

The context window is how many tokens the model can see at once: the prompt, the conversation history, any documents you pasted in, and its own reply so far. Early GPT models had about 2,000 tokens. Many current models handle 128,000, and some claim a million or more.

Here’s the catch. Attention compares every token with every other token. With n tokens, that’s n × n scores, in every head, in every layer.

attention scores ~ n²
  2,000 tokens   ->         4,000,000 pairs
 32,000 tokens   ->     1,024,000,000 pairs
128,000 tokens   ->    16,384,000,000 pairs

Double the context and the attention work goes up four times. It’s the same reason nobody builds a full-mesh VPN across thousands of sites. Ten sites in a full mesh is 45 tunnels. A thousand sites is nearly half a million. That’s why we use hub-and-spoke and SD-WAN overlays. The AI world has its own tricks for this (sparse attention, sliding windows, clever kernels like FlashAttention that cut down on memory traffic), but the basic quadratic pressure is real.

The KV cache

When a model generates text, it produces one token at a time. Each new token needs to attend to every token before it. Recomputing all the earlier keys and values from scratch for every new token would be wildly wasteful, so the model keeps them around. That’s the KV cache.

It saves a lot of compute and eats a lot of memory. Rough size:

KV cache bytes ~ 2 (K and V) × layers × tokens × d_model × bytes_per_number

For a GPT-3-sized model (96 layers, d_model 12,288, 16-bit numbers), that’s about 4.7 MB per token. A 2,000-token conversation is around 9 GB of cache, on top of the model itself. Newer models use tricks like grouped-query attention to shrink this a lot, but it’s still one of the big reasons long contexts are expensive. It’s basically a routing table that grows with every packet you forward, and it has to live in fast memory.

Next-token prediction, sampling and temperature

At the top of the stack, after the last layer, the model turns the final vector for the current position into a score for every token in its vocabulary. Those raw scores are called logits. Softmax turns them into probabilities:

p(token_i) = exp(z_i / T) / sum over j of exp(z_j / T)

That T is temperature.

T near 0: the top choice dominates. Output is predictable and repetitive.
T = 1: you sample from the model’s actual distribution.
T > 1: the distribution flattens out. More creative, more random, and eventually more nonsense.

Then the model picks a token (by sampling, often with filters like top-k or top-p that cut off the long tail of unlikely options), appends it to the text, and runs the whole thing again for the next token. Over and over until it hits a stop token or a length limit. This is called autoregressive generation.

Ham operators will get this right away. Temperature is your squelch and filter settings. Crank it down too far and you only hear the strongest station, every time. Open it way up and you hear everything, including a whole lot of noise. Somewhere in between is where the conversation is.

Which, if you’ve been counting, is Shannon’s 1948 parlor game, done with a vocabulary of a hundred thousand tokens and a context of a hundred thousand tokens, by a function with hundreds of billions of knobs.

Training vs. inference

Two very different jobs.

Training is where the parameters get set. For an LLM it happens in stages:

Pretraining. Feed the model a staggering amount of text (trillions of tokens for current frontier models) and have it predict the next token over and over. Every miss produces a gradient. Every gradient nudges the weights. This is where almost all of the compute goes and where the model picks up grammar, facts, reasoning patterns and plenty of junk.
Fine-tuning. Train further on curated examples of the behavior you want, like answering questions helpfully or following instructions.
Preference tuning, including RLHF. Reinforcement learning from human feedback. People compare pairs of model responses and pick the better one. Those choices train a separate reward model, and the main model is then tuned to produce responses the reward model scores highly. This is a big part of why a raw text-predictor becomes something that acts like an assistant. There are newer variants now (DPO and friends) that skip some steps, but the idea is the same.

Inference is using the trained model. The weights are frozen. You feed in a prompt and it generates tokens. Much cheaper per request than training, but it happens billions of times, so it adds up fast.

There’s a handy rule of thumb from the scaling-law literature (Kaplan et al., 2020) for the compute involved:

Training FLOPs   ~ 6 × N × D     (N = parameters, D = training tokens)
Inference FLOPs  ~ 2 × N         per generated token

The 2 is one multiply plus one add per parameter. The 6 for training covers the forward pass plus the backward pass, which costs about twice the forward. It ignores attention overhead and some other stuff, so treat it as back-of-envelope, but it’s a good envelope.

Kaplan’s team also showed that model quality improves in a smooth, predictable way as you scale up parameters, data and compute together. That’s a big deal. It meant labs could predict that spending ten times more compute would buy a better model, before they spent it. Later work (the “Chinchilla” paper from DeepMind in 2022) adjusted the recipe toward training on more data relative to model size. Either way, the message was the same: bigger works, as long as you can afford to compute it.

As long as you can afford to compute it. That’s the whole game.

The hardware story

Now we get to the part I really wanted to write about.

Moore’s law did the boring heavy lifting

Gordon Moore observed back in 1965 that the number of transistors you could cheaply put on a chip was doubling at a steady pace. He later settled on roughly every two years. That held, more or less, for about five decades. Doubling every two years for fifty years is 25 doublings, which is a factor of about 30 million.

For most of that run, CPUs got faster at doing one thing at a time, quickly. Clock speeds climbed from a few megahertz to a few gigahertz. Then, around the mid-2000s, they hit a heat wall and stopped climbing much. The industry pivoted to more cores instead of faster cores.

CPUs are great at complex, branchy, unpredictable work. Neural networks don’t need that. They need the same simple multiply-and-add done billions of times, in parallel. CPUs were the wrong tool, and for a long time they were the only tool.

GPUs: built for games, perfect for math

A graphics card has a different job. To draw a frame, it has to work out the color of millions of pixels, and every pixel runs roughly the same little program. So GPUs were built with hundreds, then thousands, of simple cores doing the same math on different data at the same time.

Sound familiar? That’s matrix multiplication. Embarrassingly parallel.

Researchers noticed in the early 2000s and started tricking GPUs into doing general math by disguising it as graphics. It was painful. Then in 2007 NVIDIA released CUDA, a way to program the GPU directly for general computing. That was the door opening. Suddenly a grad student with a gaming card could run parallel math that used to need a cluster.

AlexNet: the moment

In 2012, Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton entered the ImageNet image recognition competition with a deep convolutional network that came to be known as AlexNet. They trained it on two NVIDIA GTX 580 gaming cards with 3 GB of memory each, over roughly five to six days. It won by a huge margin: about 15% top-5 error versus about 26% for the runner-up.

The network design wasn’t wildly new. Convolutional networks go back to the late ’80s and ’90s. Backprop was from 1986. What made the difference was a big labeled dataset (ImageNet), a few smart tricks, and the ability to actually train a large network on consumer GPUs in under a week. The field turned on a dime after that.

Tensor cores, HBM and the bandwidth problem

Once it was clear that AI was a major workload, chip makers started building specifically for it.

Tensor cores arrived in NVIDIA’s Volta generation (2017). They are dedicated units that do small matrix multiply-and-accumulate operations in a single step, at reduced precision. Instead of a core doing one multiply-add, a tensor core chews through a little block of a matrix at once. Neural networks turned out not to need full 32-bit precision for most of their math, so 16-bit formats (FP16 and BF16) and later 8-bit formats became standard. Half the bits, roughly double the throughput, and much less memory traffic.

The math units then got so fast that they spent a lot of time waiting on memory. This is the part that will make every network engineer nod. Your compute is the CPU on the router. Your memory bandwidth is the link. It doesn’t matter how fast the router can forward if the pipe is a T1.

Enter HBM, high-bandwidth memory: stacks of DRAM sitting right next to the GPU die with an extremely wide connection. An NVIDIA H100 (SXM version) has about 3.35 terabytes per second of memory bandwidth. For reference, a decent late-’90s PC had memory bandwidth around one gigabyte per second, give or take. That’s a gap of thousands of times.

Bandwidth matters most for inference. To generate each token, the model has to read essentially all of its weights from memory. A 70-billion-parameter model at 16 bits is about 140 GB of weights. At 3.35 TB/s, just reading them once takes around 40 milliseconds. That’s your floor per token for one request on one GPU (if the model even fit on one, which at that size it doesn’t). Token generation is very often memory-bound, not compute-bound. It’s a bandwidth problem. I like this part. It’s my kind of problem.

NVLink and clusters

Even the biggest GPU isn’t enough for frontier models. So you wire many of them together. NVIDIA’s NVLink connects GPUs inside a server at hundreds of gigabytes per second each. Between servers, you get InfiniBand or high-end Ethernet fabrics. Modern training clusters run into the tens of thousands of GPUs, and the network between them is a first-class design problem. Collective operations like all-reduce, where every GPU has to share its gradients with every other GPU on every training step, live or die on the fabric.

Yes, there is a networking team quietly keeping the AI revolution alive. Somebody has to.

Cheap storage and an internet full of text

The last piece is data. Bengio in 2003 trained on corpora with millions to roughly a dozen million words. Current models train on trillions of tokens. That’s only possible because the internet created a staggering amount of text, and because storing and moving petabytes became affordable. When I started out, a gigabyte was a big hard drive. Now a petabyte is a purchase order.

Back-of-envelope: what if we tried this in 1999?

Let’s do the math. I’ll round hard and stay conservative, so if anything these numbers are generous to the old hardware.

The 1999 desktop. A good late-’90s PC had a Pentium III running somewhere around 500 MHz to 1 GHz. With its SIMD extensions, peak floating-point throughput was in the neighborhood of 1 to 2 GFLOPS, and real code usually did a lot worse. I’ll be generous and say 1 GFLOPS sustained. 10⁹ operations per second. RAM was typically 64 to 256 MB. A hard drive might stream about 20 MB per second if you were lucky.

The modern GPU. An NVIDIA H100 SXM is rated at roughly 1,000 TFLOPS of dense BF16 tensor throughput (about 990, before any sparsity tricks). That’s 10¹⁵ operations per second. Real training runs typically achieve something like 30 to 50% of peak. Call it 4 × 10¹⁴ useful FLOPS.

That’s about 400,000 times the 1999 desktop at real-world utilization, on a single chip. And nobody uses a single chip.

Generating text with a modest 7B model

Take a 7-billion-parameter model, small by today’s standards. It runs on a decent gaming PC.

FLOPs per token ~ 2 × 7×10⁹ = 1.4×10¹⁰
At 1 GFLOPS:     1.4×10¹⁰ / 10⁹ = 14 seconds per token

Fourteen seconds a token. Already painful. But that assumes the weights fit in memory. They don’t. Seven billion parameters at 16 bits is 14 GB. Your 1999 PC has maybe 128 MB of RAM. So for every single token, you’d have to stream the entire model off the hard drive:

14 GB / 20 MB/s ~ 700 seconds ~ 12 minutes per token

A typical answer of a few hundred words is something like 500 tokens.

500 tokens × 12 min ~ 6,000 min ~ 4 days

Four days to get one answer to “explain BGP to me like I’m five.” A long reply would take a couple of weeks. And that’s assuming you somehow had 14 GB of disk free, which in 1999 was a lot of disk.

On a modern GPU, the same model spits out dozens of tokens per second, sometimes over a hundred. I’ve had my dial-up connection take longer to load a single JPEG.

Training GPT-3

Now training. GPT-3 (2020) had 175 billion parameters and was trained on about 300 billion tokens.

Training FLOPs ~ 6 × N × D
              ~ 6 × 1.75×10¹¹ × 3×10¹¹
              ~ 3.15×10²³ FLOPs

On the 1999 desktop at 1 GFLOPS:

3.15×10²³ / 10⁹ = 3.15×10¹⁴ seconds
One year ~ 3.15×10⁷ seconds
=> roughly 10,000,000 years

Ten million years. On one machine. If you’d started when the earliest human ancestors were still figuring out walking, you’d be… still not done. And again, that ignores the fact that the model’s weights alone (350 GB at 16 bits, more with training state) would need thousands of 1999 hard drives.

Give it a thousand of those machines, perfectly networked, with zero overhead (which is a fantasy): ten thousand years.

On modern hardware at about 4 × 10¹⁴ effective FLOPS per H100:

3.15×10²³ / 4×10¹⁴ ~ 7.9×10⁸ GPU-seconds ~ 25 GPU-years
Spread across 1,000 GPUs ~ about 9 days

(GPT-3 itself was trained on older V100 GPUs, so it took longer and used more of them. The arithmetic is the point.)

From ten million years to under two weeks. That isn’t a better idea. That’s a better machine.

And GPT-3 is old news. Frontier models since then are widely estimated to use tens to hundreds of times more training compute. Those estimates are fuzzy, because the labs don’t publish everything, so I’ll just say: a lot more.

Fair is fair: the ideas mattered too

I don’t want to oversell this. I’ve sat through enough vendor pitches to know that “it’s all about the hardware” is exactly what a hardware vendor would say.

The transformer was a genuine breakthrough, and here’s the interesting twist: part of why it won is that it fits the hardware so well. Ditching recurrence meant you could process every token in a sequence in parallel during training. It turned language modeling into a giant pile of matrix multiplications, which is exactly what GPUs eat for breakfast. The idea and the hardware co-evolved.

Scaling laws mattered because they gave people the confidence to spend real money. RLHF and instruction tuning mattered because they turned a raw autocomplete engine into something you’d actually want to talk to. Tons of smaller tricks (better normalization, better optimizers, better data cleaning, mixed precision) each added their piece. People worked hard on all of it, and the credit is deserved.

But look at the timeline again:

1948  Shannon: predict the next letter/word (n-grams)
1966  ELIZA: rule-based conversation
1986  Backpropagation popularized
1995  A.L.I.C.E. / AIML: handcrafted conversation at scale
1997  LSTM: neural nets for sequences
2003  Bengio: neural language model + word embeddings
2007  CUDA: GPUs become programmable for general math
2012  AlexNet on two gaming GPUs
2017  Transformer; tensor cores ship
2020  GPT-3; scaling laws published
2022+ ChatGPT and the flood

The foundations were in place by 2003. The explosion came only after the hardware arrived. When the ideas waited around for decades and the breakthroughs landed right when the compute showed up, I think that tells you which one was the bottleneck.

It’s like the internet itself. Packet switching was worked out in the ’60s. TCP/IP was standardized in the early ’80s. But the web didn’t eat the world until bandwidth got cheap and ubiquitous. Nobody would say the protocol didn’t matter. But nobody streams 4K video over a 14.4 modem, either.

Is this the T-1000? About the Skynet thing

Before I sign off, I have to talk about the elephant in the server room.

A lot of people are scared of this stuff. Not “it might take my job” scared, though that’s real too. I mean Skynet scared. The fear goes like this: we’re building something smarter than us, it will wake up one day, decide humans are the problem, and we’ll end up hiding in the ruins while a liquid-metal T-1000 walks through the wall. I grew up on those movies. I get why the image sticks.

And I don’t want to cheap-shot it, because the serious version of the worry isn’t really about movie robots. It goes more like this. These systems keep getting more capable, faster than almost anyone predicted. Their builders can’t fully explain what’s happening inside the weights. Labs are racing each other. Models are being hooked up to tools, code execution, the internet and real money. If something with that much capability ends up pursuing a goal that doesn’t match what we wanted, even a little, it could do serious damage before anyone catches it. Some well-respected researchers take that seriously, and they’re not cranks.

That’s the strongest version as I understand it. Now let me take apart the part that drives most of the panic, the idea that there’s a someone in there. A sentient being with its own wants, waiting for its chance.

Go back to everything we just walked through.

What’s actually in the box

An LLM is a very large set of numbers, the weights, plus a fixed recipe for doing matrix math with them. You hand it a sequence of tokens. It runs that sequence through the layers and produces a probability for every possible next token. One gets picked. It’s appended. The whole thing runs again.

That’s it. That’s the whole lifecycle.

At inference time, which is every time you’re actually talking to one, the weights are frozen. The model is not learning from your conversation. It isn’t stewing on what you said last week. It isn’t changing itself in the background. If you ran the same prompt with the same settings and the same random seed, you’d get the same answer. Between your requests, nothing is running. There’s no little process in the corner of the data center thinking about you. The GPU memory gets handed to the next customer.

Its entire “world” is the context window. Whatever tokens are in front of it right now are all it has. When the session ends, that context is gone. Products that seem to “remember” you are doing it with plumbing outside the model: a database of notes or past chats that gets pasted back into the context next time. It’s like the difference between a router and its config. The router doesn’t remember your network out of affection. Somebody saved the config to NVRAM.

No drives, no goals, no hunger

Every living thing we know of that “wants” anything got that way because of survival pressure. You have a metabolism. You get hungry, tired, scared. Evolution spent a few billion years wiring in the drive to keep existing, because everything that lacked it didn’t stick around.

An LLM has none of that. No metabolism. No body to protect. No fear of being shut off, because there is no ongoing experience to interrupt. It wasn’t optimized to survive. It was optimized to predict the next token in human text, and then tuned to be helpful according to human raters. That’s the only pressure that ever shaped it.

So when a model writes “I don’t want to be turned off,” that’s worth understanding properly. It has read a huge amount of human writing, including decades of science fiction about AIs that don’t want to be turned off. Put it in a scenario that looks like that story and it will predict the words that tend to come next in that story. That’s the model doing exactly what it was built to do. It’s a mirror. Our fears, written down a million times, reflected back. I’m not dismissing that it can be unsettling to read. But the words aren’t a report from an inner life.

Same with “reasoning.” Modern models can work through a problem step by step in text, and that genuinely makes them better at a lot of tasks. But each step is still produced the same way: next token, given the tokens so far. There is no separate mind standing behind it deciding things out of its own original thought. The “decision” is a probability distribution, shaped by training, sampled with a temperature setting someone picked.

What about “agents”?

This is where I see the most confusion, so it’s worth slowing down.

You’ve heard about AI agents that can browse, write code, send email, book things. That sounds like the model acting on its own. Look under the hood and it’s a loop that humans wrote:

while task_not_done:
    output = model(context)          # predict tokens
    if output asks for a tool:
        result = run_tool(output)    # code a person wrote and allowed
        context += result            # paste the result back in
    else:
        break

The model produces text that says “call the search tool with these words.” Ordinary software, written and permissioned by people, actually runs the search, then feeds the result back in as more tokens. The model’s “autonomy” is exactly as wide as the tools and permissions somebody wired up. Give it read-only access and it can only read. Give it the keys to production and, well, now we’re talking about a different problem.

Which brings me to the part I actually worry about.

The real risk is us

I’ve spent my career around networks, and here’s something every network engineer learns the hard way. A routing table doesn’t have feelings. It has no malice. And one bad route, pushed to the wrong place at the wrong time, can black-hole traffic for an entire region. Big outages have come from a single fat-fingered config. Nobody’s router “decided” to take down the network. A powerful system did exactly what it was told, at full speed, everywhere at once, and the instruction was wrong.

That’s my mental model for AI risk.

A chainsaw isn’t malicious, either. It’s also not something you hand to someone who has never used one and then walk away. The danger isn’t that the saw wakes up. It’s that it’s very good at cutting, and it doesn’t care what it’s cutting.

So the risks I take seriously look like this. People trusting confident output without checking it, in medicine, law, code, and yes, network configs. Automation wired straight into systems that matter, with no human checkpoint and no rollback plan. Bad actors using these tools to write phishing at scale, generate convincing fakes, or speed up attacks. Companies deploying systems they don’t fully understand because the competitor across the street already did. And models trained on our data quietly repeating our biases with a straight face.

None of that requires a sentient machine. All of it requires is a powerful tool and humans being human.

I want to be fair to the safety folks here, too. A lot of what gets called “AI safety” is exactly this kind of work: testing what models will and won’t do, finding failure modes before customers do, building guardrails, studying what’s going on inside the weights, and arguing for sensible limits on how much access these systems get. That’s good engineering. It’s change control and least-privilege for a new kind of system. I’m all for it. And to be honest about my own limits, nobody can promise what systems built ten or twenty years from now will look like, and some serious people think the long-term risks deserve real attention. That’s a fair debate. What I’m pushing back on is the idea that the chatbot you used this morning is a trapped mind plotting its escape.

The good news is that we already know how to handle powerful systems that can fail at scale. We’ve been doing it with networks and power grids for decades. You limit permissions. You keep a human in the loop for anything that can’t be undone. You log everything. You test in a lab before production. You have a rollback plan. You don’t give a new system enable access on day one just because it passed a demo.

There’s no T-1000 coming through the wall. There’s a very powerful autocomplete engine, and a lot of people in a hurry to plug it into things. Worry about the second part.

Signal and noise

I’ll leave you with the thing that keeps rattling around in my head.

Shannon’s whole career was about separating signal from noise. How much real information is in a message, and how do you get it through a noisy channel intact? Next-token prediction is that same question turned around. A model that can predict text well has captured a lot of the structure, the signal, in how people write and think. It turns out that predicting the next token well enough requires picking up an awful lot about the world along the way.

None of that means these models are magic, or conscious, or always right. They’re very large statistical machines, trained on our words, running on silicon that would have looked like science fiction when I was waiting for a modem handshake. They make confident mistakes. They need checking. Treat their output the way you’d treat a sketchy signal report from a station you don’t know: useful, worth listening to, verify before you bet anything on it.

But the next time someone tells you AI came out of nowhere in 2022, you can tell them the truth. The machines have been trying to talk since before most of us were born. We just finally built them a big enough pipe.

73.

The Machines Were Always Trying to Talk. The Hardware Finally Let Them.