What Actually Happens When You Hit Enter on an AI Prompt

You type a question into a chatbot, you hit Enter, and about a second later words start appearing on your screen like somebody on the other end is a very fast typist. (Nobody is typing. There is no tiny person in the data center. I promise.)

So what actually happened in that second?

In The Machines Were Always Trying to Talk, I walked through what goes on inside the model: tokens, attention, the KV cache and why GPUs matter. In Why AI Is Secretly a Networking Problem, I went into the fabrics that train these things. This post is the trip in between. It’s the AI prompt lifecycle traced the way a network engineer traces a packet: from your keyboard, across the internet, through a stack of front doors, into a GPU cluster, and back to your screen one token at a time.

If you’d rather watch first, here’s the whole trip in about 3 minutes.

Now let’s slow it down.

The prompt we’re going to follow

Every good packet capture needs a test case, so here’s ours:

“Why does my BGP session keep flapping?”

(Yes, I picked a networking question. Old habits.)

When you hit Enter, the chat app doesn’t send those words by themselves. It wraps them in an HTTPS request to the provider’s API, usually a POST with a JSON body. Every vendor’s format is a little different, but the shape looks something like this:

POST /v1/chat HTTP/2
Host: api.example-ai.com
Authorization: Bearer <your API key>
Content-Type: application/json

{"model": "some-model",
 "stream": true,
 "messages": [{"role": "user",
   "content": "Why does my BGP session keep flapping?"}]}

Two details matter for the rest of this post. The Authorization header is how the provider knows who to bill (and who to rate limit). And that little "stream": true flag is the reason the answer shows up word by word instead of all at once; we’ll come back to it at the end, because it changes how every box in the path has to behave.

Step 1: DNS finds the front door

Before your laptop can send anything, it needs an IP address for the API’s hostname. Most of the time the answer is already sitting in a cache somewhere between you and the resolver, so this step costs next to nothing.

When it isn’t cached, the resolver walks the DNS tree and comes back with an answer. For big providers, that answer is often an anycast address: the same IP announced from many locations, with BGP steering you to whichever one is “closest.” Closest in BGP terms, mind you, which any network engineer will tell you is not always closest in miles. (AS path length has never once asked me how far away Dallas is.)

The point is that the IP you get back usually isn’t a GPU server. It’s the front door of a much bigger building.

Step 2: TLS 1.3 sets up the secure tunnel

Next comes the handshake. If you’re on HTTP/2, that means a TCP handshake first, then TLS. (HTTP/3 runs over QUIC, which folds the transport and TLS handshakes together.)

TLS 1.3 needs one round trip. Your client sends a ClientHello that already includes its key share. The server answers with a ServerHello, its certificate and a Finished message, and per RFC 8446 everything after the ServerHello is already encrypted. The client sends its own Finished, and the HTTP request rides right behind it.

Two extensions in that ClientHello do a lot of quiet work. SNI (server name indication) tells the edge which hostname you want, so it can present the right certificate. ALPN lets the two sides agree on HTTP/2 before any HTTP is spoken.

I got dial-up when I was about 14, so I remember when a “handshake” was a screeching noise coming out of a modem. TLS 1.3 is the same idea minus the screech.

What about 0-RTT? TLS 1.3 does allow “early data” on a resumed session, sent before the handshake finishes. The RFC is blunt about the catch: early data isn’t forward secret and has no guarantee against replay. For a prompt that costs real money to run, a replayed request is not a bug you want to explain to finance.

A quick worked example. Say you’re 20 ms round trip from the nearest edge (I’m making that number up to keep the math easy). A cold connection costs about one round trip for TCP and one for TLS before your request is even on the wire: roughly 40 ms of pure physics. The good news is that chat apps keep the connection open, so your second prompt skips all of it. Connection reuse is the cheapest latency win in this whole post.

Step 3: The edge and the load balancer

That anycast IP lands you at an edge location. In most large deployments, the edge is where your TLS session actually terminates, close to you, and where junk traffic gets filtered before it gets anywhere expensive. From there, your request rides inward over the provider’s own backbone toward a region that hosts the model.

Then a load balancer picks a healthy backend with room to spare. Nothing exotic so far; this is the same pattern that serves web pages.

Here’s where AI traffic starts to get weird, though. A normal web request is in and out in milliseconds. A streamed AI response can keep a connection open for seconds or minutes while the model thinks and types. Every hop in the path, right down to that proxy nobody remembers deploying, has to be fine with a long-lived response that trickles out a few bytes at a time. Remember that; it bites later.

Step 4: The API gateway checks your ID

Before your prompt gets anywhere near a GPU, an API gateway does the bouncer work:

• Authentication. Is that API key real, and whose is it?
• Rate limiting. Are you over your quota? Many AI providers count tokens as well as requests, because one giant prompt can cost more than a hundred small ones.
• Routing. Which model did you ask for, and which pool of GPUs serves it?
• Queueing. If every GPU in that pool is busy, your request waits in line.

That last one is the step nobody sees and everybody feels. When demand outruns supply, the queue is where your “AI is slow today” experience usually comes from. I wrote about the bigger picture of that squeeze in AI Inference Rationing.

Step 5: Tokenization turns your words into numbers

The model doesn’t read words. It reads tokens, which are chunks of text mapped to integer IDs. (I covered the basics in the first post, so here I just want to show you a real one.)

I ran our test prompt through OpenAI’s open source tokenizer, tiktoken, using the o200k_base encoding. Here’s what came out:

Token	ID
Why	13903
·does	2226
·my	922
·B	418
GP	29864
·session	6223
·keep	3357
·fl	1548
apping	5041
?	30

(The dots stand for leading spaces, which are part of the token.)

Ten tokens for 38 characters, which lines up nicely with NVIDIA’s rule of thumb that a token is about four characters of English. And notice that “BGP” got chopped into ” B” and “GP.” The tokenizer has clearly never been paged at 2 AM over a flapping peer. Different models use different tokenizers, so the exact split varies, but the idea is the same everywhere.

From here on, your prompt is just a list of ten numbers.

Step 6: Prefill reads the whole prompt at once

Now the GPU finally gets involved. Serving a prompt happens in two very different phases, and understanding the split explains most of what you feel as a user.

The first phase is prefill. The model processes all ten prompt tokens in one parallel pass. In NVIDIA’s words, it’s a highly parallel matrix-matrix operation that keeps the GPU busy; this is the phase where all those GPU cores actually earn their keep. Along the way, it builds the KV cache: the stored keys and values for every prompt token, so the model never has to recompute them.

Prefill is what stands between you and the first word of the answer. NVIDIA’s benchmarking docs define time to first token (TTFT) as including the request queuing, the prefill and the network latency, and they note that longer prompts mean a longer TTFT. Paste a 50-page PDF into a chat and you’ll feel exactly that.

Prefix caching is the shortcut. A lot of prompts start the same way. Think of the system instructions an app sends with every request, or that long document you keep asking follow-up questions about. vLLM, a popular open source serving engine, handles this by hashing KV cache blocks based on their tokens plus everything before them. If a new request starts with the same blocks, the server reuses them instead of running prefill again. The vLLM docs note that public endpoints use this widely, and they add a per-request salt option so one tenant can’t probe another’s cache with timing tricks.

To a network engineer, this is fast switching versus process switching. The first packet of a flow takes the slow path; everything after it hits the cache.

Step 7: Decode writes the answer one token at a time

The second phase is decode, and it’s a completely different animal.

The model generates one token, adds it to the KV cache, then runs again to generate the next one, and again, and again. Every step reads the model weights and the whole KV cache from GPU memory to produce a single token. NVIDIA describes decode as a matrix-vector operation that is memory-bound: the GPU’s math units mostly sit waiting on memory. The vLLM paper makes the same point and says decode is responsible for most of the latency.

Here’s what that memory looks like with real numbers. NVIDIA’s example: Llama 2 7B at 16-bit precision has about 14 GB of weights, and the KV cache for a single 4,096-token sequence is about 2 GB. The vLLM paper puts OPT-13B at 800 KB of KV cache per token, which works out to as much as 1.6 GB for one 2,048-token request.

When we moved the family to Georgia, I drove a 26-foot manual moving truck while blasting Whitesnake. Now picture that truck in stop-and-go traffic. The engine isn’t the problem; getting the whole load moving again, every single time, is. Decode is that truck: each new token means dragging the weights and the KV cache through memory one more time.

Batching: why one GPU serves a lot of people at once

If decode reads all the weights just to make one token, the obvious fix is to make a lot of tokens per read, for a lot of users at the same time. That’s batching, and the way it’s done changed a lot in the last few years.

The old way was static batching: group some requests, run them until the longest one finishes, then start the next group. The Orca paper (OSDI 2022) pointed out the problem: requests that finished early couldn’t return, and new arrivals had to wait for the whole batch. Its fix was iteration-level scheduling, now usually called continuous or in-flight batching. The scheduler works one decode step at a time, so finished requests leave and new ones join after every step.

If you came up through networking, you already know this story. Static batching is TDM: reserved time slots that sit empty when a caller has nothing to say. Continuous batching is packet switching. Statistical multiplexing wins again.

Memory was the other bottleneck. The vLLM team measured that existing systems used only 20.4% to 38.2% of their KV cache memory for actual token data, because they reserved space for the longest possible answer up front. Their fix, PagedAttention, borrows from operating system virtual memory: it carves the KV cache into fixed-size blocks (16 tokens by default) and hands them out as needed. Less waste means more requests per GPU, and the paper reports 2 to 4 times the throughput of earlier systems at the same latency.

The tradeoff you already know from QoS

More users per GPU isn’t free. NVIDIA’s benchmarking docs spell it out: as concurrency goes up, total tokens per second rises until the system saturates, while tokens per second for each individual user falls.

Everything I know about queueing started with a Cisco Press QoS book by Michael Cavanaugh and Wendell Odom, the book that made Cav my mentor before he even knew it. The lesson carries over perfectly. You can run a link at 95% utilization or you can have low latency, and you have to pick. GPU clusters face the exact same choice; they just call it “batch size.”

Some providers go further and split prefill and decode onto separate GPU pools (disaggregated serving). The vLLM docs list two reasons: you can tune time to first token and time between tokens independently, and you stop prefill jobs from cutting in line during decode and spiking tail latency. The catch is that the KV cache has to move between the pools over the network, which is exactly the problem I covered in Why AI Is Secretly a Networking Problem.

(How the model actually picks each next token, temperature and all, is in the first post.)

Step 8: Tokens stream back over server-sent events

Remember "stream": true?

Without it, OpenAI’s docs say the whole answer gets generated before a single HTTP response goes back. You’d stare at a spinner for the full decode time. With streaming on, the server answers using server-sent events (SSE): a plain HTTP response with the content type text/event-stream that just… stays open. Each event is a few lines of text, separated by a blank line. Here’s what a single word looks like in Anthropic’s format:

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" hold"}}

Multiply that by every token, and you have the “typing” effect. (It’s not an animation. Those really are the words arriving as the model writes them.)

When I was a teenager I ran a BBS called The Hydra’s Lair on the family phone line. I tied up the line, I got grounded, and eventually the family added a second line. SSE is the same move, holding one connection open for the whole answer, except nobody gets grounded. HTTP/2 is the second phone line: MDN notes that without HTTP/2, a browser gets only six SSE connections per domain, while HTTP/2 multiplexes many streams over one connection.

Where streaming breaks (the network engineer’s section)

This is the part of the trip where our side of the house can ruin everything:

• Buffering proxies. Some reverse proxies collect a response before forwarding it. That’s great for web pages and terrible for a stream; your user sees nothing, then the whole answer at once. MDN’s own SSE example sends an X-Accel-Buffering: no header to tell nginx not to buffer.
• Idle timeouts. A model can pause while it works through a long answer. If a load balancer or firewall decides a quiet connection is dead, the stream just stops mid-sentence. SSE has a built-in answer: a line starting with a colon is a comment, which MDN suggests as a keepalive, and Anthropic’s stream includes ping events as well.
• Security inspection. Anything in the path that wants the full response body before passing it along (some DLP and inspection tools) will behave like a buffering proxy. Test it before your users do.

Hmmm. Notice that none of these are AI problems. They’re plain old networking problems that AI traffic happens to trip over.

Putting numbers on the whole trip

NVIDIA’s benchmarking docs give the cleanest way to think about it: end-to-end latency = time to first token + generation time. Here’s a back-of-the-napkin example using made-up but reasonable round numbers, just to show where the time goes:

Stage	What drives it	Example
DNS	Usually cached	about 0 ms
TCP + TLS 1.3	Round trips to the edge	about 40 ms cold, 0 ms on a reused connection
Edge, load balancer, gateway	Backbone distance and checks	a few tens of ms
Queue	How busy the GPU pool is	0 ms to “go get coffee”
Prefill	Prompt length	short for our 10 tokens, long for a 50-page PDF
Decode	Memory bandwidth and batch size	at 50 tokens per second, a 500-token answer takes 10 seconds

Everything above the decode line adds up to your time to first token. The decode line is how fast the answer “types.” That’s why a chatbot can feel snappy at the start and slow at the end, or the other way around; they’re two different bottlenecks.

What this means if you run the network

If you’re the person who gets the ticket when “the AI is slow,” here’s my short list:

• Measure TTFT and time between tokens separately. They point at different culprits: the path and the queue for the first, the GPU for the second.
• Don’t buffer text/event-stream. Check every reverse proxy, inspection device and load balancer in the path.
• Set idle timeouts with streaming in mind, and confirm keepalives actually make it through.
• Keep connections warm. Reuse beats any handshake optimization.
• Put the stable part of a prompt first if you build on these APIs. Prefix caches match from the start of the prompt, so a shared system prompt up front gives the cache something to reuse.

And if you want to see every hop for yourself, run a model at home. In Run Your Own AI at Home for the Price of a Gaming PC, the whole trip collapses to your own box: no DNS, no edge, no queue except you. If you’re weighing that against paying per token, the AI Rent vs. Own vs. Hybrid TCO model does the math.

Recap

From lowercase point a, to UPPERCASE POINT B, here’s the trip your prompt takes:

• DNS turns the API’s name into an address, usually an anycast front door.
• TLS 1.3 builds an encrypted tunnel in one round trip.
• The edge and load balancer carry you inward to a healthy region.
• The API gateway checks your key, enforces your limits and puts you in line.
• The tokenizer turns your words into numbers.
• Prefill reads the whole prompt at once and builds the KV cache.
• Decode writes the answer one token at a time, sharing the GPU with everyone else in the batch.
• SSE streams each token back over a connection that stays open the whole time.

About a second, start to finish, for the first word. Most of the magic is just good networking with a very expensive calculator at the far end.

Don’t take my word for it; the sources are below. And if you’ve ever chased a “slow AI” ticket that turned out to be a buffering proxy, I’d love to hear about it.

Keep reading

• The Machines Were Always Trying to Talk. The Hardware Finally Let Them. The first post in this series: what happens inside the model.
• Why AI Is Secretly a Networking Problem. The fabrics behind training and the KV cache transfers behind disaggregated serving.
• Run Your Own AI at Home for the Price of a Gaming PC. The same trip, minus the internet.
• AI Rent vs. Own vs. Hybrid. A free decision model for what all of this costs.
• AI Inference Rationing. What happens when the GPU queue gets long.

References

• E. Rescorla, RFC 8446: The Transport Layer Security (TLS) Protocol Version 1.3, IETF, August 2018.
• RFC 9000: QUIC, a UDP-Based Multiplexed and Secure Transport, IETF.
• NVIDIA Technical Blog, Mastering LLM Techniques: Inference Optimization, November 2023.
• NVIDIA, NIM for LLMs Benchmarking: Metrics.
• W. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023.
• G. Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models, OSDI 2022.
• vLLM documentation, Automatic Prefix Caching and Disaggregated Prefilling.
• MDN Web Docs, Using server-sent events.
• Anthropic, Streaming Messages.
• OpenAI, Streaming API responses.
• OpenAI, tiktoken (tokenizer used for the example above).