AI Inference Rationing: Is This the Day the Music Died?

What happens when the world wants more AI than the world can physically power? Not in a sci-fi way. In a boring, utility-bill, waiting-on-a-transformer way. If the frontier labs can’t build data centers fast enough, AI inference rationing stops being an annoying edge case and becomes the normal way we buy tokens.

So is that the day the music died? Does the genie go back in the bottle? I don’t think so. But I do think the all-you-can-eat era we’re in right now has an expiration date. I want to lay out why, what I think happens instead, and how likely each outcome is.

Quick note: the probabilities in here are mine. Opinions built on public data, linked so you can argue with me.

AI inference rationing is already here, we just don’t call it that

If you lean on Claude, ChatGPT or Gemini hard enough, you’ve probably already run into it. In July 2025 Anthropic announced weekly rate limits for its Pro and Max plans, aimed at people running Claude Code “continuously in the background, 24/7.” Then in March 2026 it went a step further and made five-hour session limits drain faster during weekday peak hours, 5 to 11 a.m. Pacific. Anthropic’s own estimate was that about 7% of users would hit limits they wouldn’t have hit before.

That’s time-of-day pricing. Utilities have been doing it for decades.

OpenAI has had its moments too. In March 2025, when image generation went viral, Sam Altman said “our GPUs are melting” and the company put temporary limits in place. On the API side, OpenAI now sells flex processing, which is cheaper but slower and can come back with a “429 Resource Unavailable” when there isn’t capacity. At the other end there’s Fast mode, which charges a per-token premium. For GPT-5.6 Sol, that’s twice the standard rate.

Put those together and the whole playbook already exists: caps, peak-hour throttling, a discount tier you can get bumped from, and a premium lane. Anybody who’s built networks recognizes this. It’s QoS. It’s oversubscription with a priority queue.

Here’s the part people miss, though. In May 2026 Anthropic doubled Claude Code’s five-hour limits and dropped the peak-hour reduction for Pro and Max, right after signing a deal for all of SpaceX’s Colossus 1 data center. That’s more than 300 megawatts and over 220,000 NVIDIA GPUs. The rationing loosened the moment capacity showed up. So these limits aren’t really a pricing philosophy. They’re about megawatts.

The wall is made of power, packaging and memory

Power is the slowest thing to build

The IEA expects global data center electricity use to more than double by 2030, to around 945 TWh, which is slightly more than all of Japan uses today. Demand from AI-optimized data centers alone is projected to more than quadruple. In the US, data centers are on course for almost half of all electricity demand growth between now and 2030.

Zoom in on the US and it gets steeper. Lawrence Berkeley National Lab’s report for DOE had data centers at 4.4% of US electricity in 2023 and projected 6.7% to 12% by 2028. EPRI’s 2026 update goes higher: 9% to 17% by 2030, up from 4% to 5% today, and about 60% above EPRI’s own 2024 estimate.

The problem isn’t just how much. It’s where, and how fast. EPRI points out that a new 100 to 1,000 megawatt data center is like dropping in a neighborhood of 80,000 to 800,000 homes, except the data center goes up in a few years and the grid takes a lot longer. ERCOT in Texas told lawmakers in April 2026 that it’s tracking about 410 GW of large loads trying to connect, and roughly 87% of them are data centers. On the supply side, Berkeley Lab counted about 2,290 GW of generation and storage sitting in US interconnection queues at the end of 2024. Queues are wish lists, not plans, but the gap tells you which side is waiting on which.

Servers ship in months. Substations, transmission lines and turbines take years.

Chips, HBM and packaging

Even with power, you need parts. On TSMC’s July 2026 earnings call, CEO C.C. Wei said packaging capacity is “so tight that now it’s limiting my customers’ growth.” He’s talking about CoWoS, the advanced packaging that bonds GPU dies to high-bandwidth memory. Industry estimates have CoWoS capacity roughly doubling by the end of 2026, and it’s still reported as fully booked. SK hynix said in July that customer demand for HBM4 exceeds its supply.

The money

Microsoft, Alphabet, Amazon and Meta have guided to roughly $720 to $745 billion in combined 2026 capex. Not all of it buys compute. Evalueserve’s analysis notes Microsoft and Meta tied about $35 billion of their raised guidance to component and memory pricing. That’s inflation, not capacity. When the inputs cost more, the same dollar buys fewer tokens.

The other side of the ledger: tokens keep getting cheaper

If this post stopped here, you’d figure we’re headed straight into a wall. We’re not, or at least not cleanly, because the cost of any given level of AI keeps falling at a ridiculous rate.

Epoch AI’s September 2026 report estimates the price of a given level of AI performance has dropped about 47% per quarter since 2023. That’s roughly 13x per year, faster than DNA sequencing, compute or batteries ever got cheaper. Stanford’s 2025 AI Index found the cost of GPT-3.5-level performance fell more than 280-fold between November 2022 and October 2024.

A lot of that comes from engineering, not just new silicon. Mixture of experts is why DeepSeek-V3 has 671 billion parameters but only activates 37 billion per token. Speculative decoding lets a small model draft tokens that the big model checks in parallel. Google’s original paper showed 2x to 3x speedups with identical output. Quantization and distillation shrink models so they fit on less hardware.

Two catches, though. First, Epoch found the price drop slows as a capability level ages. It’s about 75x per year right when a level of performance is state of the art, and about 4.7x per year two years later. Second, and this is the big one, cheaper doesn’t mean less.

Satya Nadella said it out loud in January 2025: “Jevons paradox strikes again!” Make something cheaper and people use way more of it. Google’s numbers show what that looks like. At I/O 2026, Sundar Pichai said Google went from 9.7 trillion tokens a month two years earlier, to about 480 trillion a year earlier, to over 3.2 quadrillion a month. He also said many companies were “already blowing through their annual token budgets, and it’s only May.”

Agents make this worse. A chatbot answers when you ask. An agent works all night and burns tokens the whole time, which is exactly what pushed Anthropic to add weekly limits.

So you’ve got two curves racing. Efficiency pulls the cost per token down fast. Demand pushes total tokens up even faster. And power and packaging set a speed limit on how much physical capacity can exist in any given year. The real question is which curve wins, and where.

Four ways this plays out

Here are the four scenarios I think are plausible through about 2030. I’ve framed each one as the picture that best describes the world at that point, so they don’t overlap and the odds add up to 100%. Again, these are my estimates.

Scenario 1: Abundance wins (20%)

Efficiency gains, new chips and the capex wave outrun demand. Today’s limits turn out to be growing pains, like the peak-hour throttle Anthropic rolled back for Pro and Max about six weeks after adding it. Flat-rate plans stay generous, API prices keep falling, and nobody thinks much about tokens.

Why I don’t rate it higher: Jevons. Every time capacity frees up, agents, video and reasoning models soak it right back up. And power plants don’t follow Moore’s law.

Scenario 2: The metered frontier (30%)

Rationing becomes the business model. Rate limits, peak-hour pricing, priority lanes and committed-capacity contracts become normal, the way they already are for electricity and reserved cloud GPUs. Most work still runs on big centralized frontier models, but you pay a lot more for them when everybody else wants them too. Consumer flat-rate plans get tighter or more tiered.

Every piece of this already exists. The reason it isn’t my top pick is that customers won’t just eat the surge price forever. They’ll engineer around it, which gets us to scenario 3.

Scenario 3: Tiered and distributed (40%)

This is the one I’d bet on. Frontier capacity stays scarce and priced like it, so the industry splits up the work. Routine, high-volume stuff like classification, extraction, summarization, routing and a lot of agent steps moves to small language models running on-prem, at the edge or right on the device. The frontier models get saved for the hard problems: deep reasoning, novel work and long-horizon tasks.

The evidence already points this way. NVIDIA researchers argued in 2025 that small language models are the future of agentic AI. They estimated a 7B model is 10x to 30x cheaper to serve than a 70B to 175B model, and that 40% to 70% of the model calls in the open-source agents they looked at could be handled by specialized small models. Gartner predicts that by 2027 organizations will use small, task-specific models three times more than general-purpose LLMs. Apple already gives developers direct access to an on-device model of about 3 billion parameters. Even EPRI has a new collaboration studying smaller data centers designed for distributed inference.

Networking went through the same thing. You don’t backhaul every packet to the core. You push intelligence out to the edge and keep the core for what actually needs it.

Scenario 4: The wall, or the bust (10%)

Something breaks hard. Grid limits, a supply shock in packaging or memory, or a financing pullback freezes the build-out, and access stays rationed for years with real economic pain. Or the capex bubble pops and building just stops.

Why only 10%: too many players with too much money and too many workarounds, from on-site generation to long-term nuclear deals. Also, a financial bust doesn’t automatically mean scarcity. The telecom crash after the dot-com era left a lot of fiber in the ground that got lit up later. A GPU glut would make inference cheaper for a while, not more expensive.

Scenario	My probability	Signals to watch
1. Abundance wins	20%	Usage limits keep getting raised. API prices fall faster than usage grows. CoWoS and HBM lead times shorten.
2. The metered frontier	30%	Peak pricing and priority premiums spread to more providers. Flat-rate plans get capped or split into tiers. Reserved capacity contracts become standard.
3. Tiered and distributed	40%	Model routers show up by default in enterprise AI stacks. Small model and on-device usage grows faster than frontier usage. Edge and on-prem inference hardware sales climb.
4. The wall, or the bust	10%	Grid operators delay or pause large-load connections. Hyperscalers cut capex guidance together. Long waitlists or sustained capacity outages for API access.

So, is this the day the music died?

No. The genie isn’t going back in the bottle. You can’t un-invent a 3 billion parameter model that runs on a phone, and the open-weight models people have already downloaded aren’t going anywhere.

What dies is the free-for-all. The flat $20 a month for what feels like unlimited frontier compute. The habit of sending every task, no matter how trivial, to the biggest model available. That’s not the music dying. That’s the house lights coming up and the bar charging full price for drinks again.

My predictions for how wide-scale adoption reshapes AI

These are forward-looking opinions, so take them that way. Here’s where I land.

Prediction 1: Routing becomes the default architecture

In a few years, sending every request to one frontier model will look as dated as running a flat network with no VLANs. Most serious AI stacks will put a router in front that picks the model by difficulty, data sensitivity, latency and cost. The frontier model becomes the escalation path, not the default. Pichai basically pitched this at I/O, telling companies they could save a lot of money by mixing Flash with other frontier models.

Prediction 2: Tokens get metered like a utility

Expect off-peak discounts, demand charges, priority premiums and reserved capacity, just like power. We already have flex tiers, a 2x priority lane and peak-hour session limits. I think an AI bill in 2030 will look a lot like a commercial electric bill.

Prediction 3: Enterprises rediscover TCO discipline

When API price and availability get less predictable, the rent vs. own math changes. If you can’t count on capacity at a stable price, owning some of your own inference starts to look a lot better, especially for steady, routine or sensitive work. That’s a big reason I built my free AI Rent vs. Own vs. Hybrid TCO model. Raise the API price, or lower the expected annual price drop, and watch the answer move. My bet is a lot of organizations end up hybrid. Not because it’s trendy, but because the spreadsheet says so.

Prediction 4: Power becomes the real moat

Models are commoditizing fast. Epoch’s data says a given level of performance gets dramatically cheaper within a year or two of debuting. What doesn’t commoditize fast is energized, grid-connected capacity. The companies that lock up megawatts, interconnection rights and long-term power deals will hold an advantage that a better model only buys you for a few months. When AI labs start announcing compute deals in gigawatts, that tells you where the real competition is.

Prediction 5: Smaller gets smarter, and the edge gets busy

The most-used AI in 2030 won’t be the smartest one. It’ll be the small, cheap, fast model running on your laptop, your phone, your car or a box in the wiring closet, quietly handling the boring stuff. The frontier will still matter a lot. It’ll just be the specialist you call in, not the one doing every job.

The gold rush doesn’t end because we run out of gold. It ends when the picks, the shovels and the power to run them start costing what they’re actually worth. If you’re planning AI spend for the next few years, plan for that world. Think I’ve got the odds wrong? Drop a comment or find me on LinkedIn.

Keep reading

If this got you thinking, here’s what I’d read next:

• AI Rent vs. Own vs. Hybrid: A Free TCO Decision Model in Excel. Plug in your own numbers and see how the rent vs. own answer shifts when API prices and availability stop being a sure thing.
• The Machines Were Always Trying to Talk. The Hardware Finally Let Them. How AI actually works under the hood, and why the hardware this whole post is worried about was what made LLMs possible.
• How to Build Your Own Private Family Streaming Service. Rent vs. own at home. Three Raspberry Pi 5s running my family’s whole streaming setup.