Why AI Is Secretly a Networking Problem

Everybody wants to talk about the GPUs. They get the headlines and the keynote slides, and I get it. But I’ve spent more than twenty years building networks and carrying voice traffic, and every time I dig into how the giant AI clusters actually work, I keep landing in the same place.

The hard part isn’t the math. It’s the wire between the math.

AI networking is the quiet discipline holding the whole boom together. Training a frontier model means tens of thousands of GPUs acting like one enormous computer, and that only works if the network moving data between them is fast, predictable and, ideally, never drops a packet. One congested link can slow down thousands of GPUs at once. That’s not me being dramatic; it’s in the engineering papers, and I’ll link them as we go.

If you read my post on how AI really works, you know my take: the hardware finally caught up with the math. This is the part of the hardware story nobody puts on the poster. So let’s go from lowercase point a, “what does the network have to do with AI?”, to UPPERCASE POINT B, where you could hold your own in a conversation about RoCE, InfiniBand and rail-optimized fabrics.

Why GPUs can’t just work alone

A big language model doesn’t fit on one GPU. Not the weights, not the training state, not the data. So the work gets chopped up and spread across hundreds or thousands of GPUs, and each one handles a slice. Meta’s Llama 3 paper describes training its largest model on up to 16,000 H100 GPUs using four kinds of parallelism at once (they call it 4D parallelism, which sounds like a sci-fi movie but is really just four different ways of slicing the job). The paper is long, but network folks should read the infrastructure section.

Here’s the catch. After every chunk of work, those GPUs have to compare notes. Each one has learned something slightly different from its slice of the data, and they all need to agree on the result before anybody moves to the next step.

That “compare notes” step is called a collective operation, and the most famous one is all-reduce. Every GPU contributes its numbers, they get summed (or averaged) across the group, and every GPU walks away with the same answer. Picture a conference call where nobody can hang up until everyone has read their numbers aloud and written down the total. Libraries like NVIDIA’s NCCL (pronounced “nickel”) handle this very well.

But they can’t change physics.

The math that makes network people sit up straight

NVIDIA’s own nccl-tests documentation lays out the formula for a ring all-reduce. With n GPUs, each one has to move data equal to 2(n-1)/n times the size of what’s being reduced. For 8 GPUs that’s 1.75 times the data. For 1,000 GPUs it’s basically 2 times.

Translation: every GPU has to send and receive about double the size of the data, every single step, and the step isn’t done until the slowest GPU finishes. A quick back-of-the-napkin example (my numbers, purely to show the shape of it): if each GPU needs to reduce 10 GB and has a 400 Gb/s link, that link moves about 50 GB per second, so roughly 20 GB of traffic takes at least 0.4 seconds. Real systems overlap this with compute and shard things cleverly, which helps a lot. The lower bound is still set by the network.

Now multiply that by thousands of GPUs and thousands of steps. Hmmm.

The tail wags the whole dog

In 2013, Jeff Dean and Luiz André Barroso at Google wrote a paper called “The Tail at Scale.” Their example stuck with me: if a single server is slow on just 1 out of every 100 requests, and a user’s request has to collect answers from 100 of those servers in parallel, then 63% of user requests end up slow. Rare hiccups on individual boxes become the normal experience at scale.

AI training is that problem with the volume cranked to eleven. A collective operation is only as fast as its slowest participant. If one link is congested, or one switch port is flapping, or one NIC is having a bad day, every other GPU in that group just sits there waiting. Expensive silicon, idling, waiting on a packet.

Meta said it plainly in the Llama 3 paper: “Even a single straggler can slow down thousands of other GPUs, often appearing as functioning but slow communications.” Functioning but slow. If you’ve ever been on a bridge call where everything is “up” and green on the dashboard but users are still screaming, you know exactly how much fun that is to troubleshoot.

(Ask me how I know that one switch can ruin everyone’s night. Actually, don’t; I wrote about my midnight hospital IDF outage in my book, and once was enough.)

To be fair to the GPUs

I want to be honest here, because it would be easy to oversell the networking angle. In that same Llama 3 paper, Meta logged 419 unexpected interruptions over a 54-day stretch of training. GPU problems were the biggest category by far, at 58.7%. Network switches and cables accounted for 35 interruptions, or 8.4%.

So no, the network isn’t the most common thing that breaks. The point is different: when the network misbehaves, it doesn’t take out one server. It drags down a synchronized job that spans thousands of them. And the subtle failures (slow, not dead) are the ones that eat your week.

RDMA: skipping the line at the DMV

Normal networking goes through the operating system. An application hands data to the kernel, the kernel builds packets, copies things around, and eventually the NIC sends them. Fine for web traffic; way too slow and CPU-hungry for GPUs swapping gigabytes constantly.

RDMA (Remote Direct Memory Access) fixes that by letting the network card read and write memory on another machine directly, bypassing the kernel on both ends. Meta’s SIGCOMM 2024 paper, “RDMA over Ethernet for Distributed AI Training at Meta Scale”, describes it as transferring data “directly from/to application memory” with the work handled in NIC hardware. Add GPUDirect RDMA and the NIC can talk straight to GPU memory, so the data doesn’t even stop at the CPU for a coffee.

There are two big ways to carry RDMA in an AI cluster.

InfiniBand was built for this from day one. It uses credit-based flow control: a sender doesn’t transmit unless the receiver has already said it has buffer space. NVIDIA’s support documentation puts it simply: InfiniBand was designed as a lossless network, and instead of dropping packets under congestion, it just doesn’t send until there’s room. Packets are only deliberately dropped after a timeout meant to prevent deadlock.

RoCEv2 (RDMA over Converged Ethernet, version 2) takes the RDMA transport and wraps it in regular Ethernet, IP and UDP headers so it can be routed across a normal data center fabric. It even has its own IANA-registered port, UDP 4791, which is a fun thing to filter on in Wireshark.

The catch? RDMA really doesn’t like packet loss, and classic Ethernet drops packets as a way of life. Meta’s paper notes they used a go-back-N retransmit scheme for the rare drops, and that lost acknowledgement packets could cause timeouts “on the order of milliseconds.” Milliseconds sound tiny. In a cluster where everyone waits for the slowest GPU, milliseconds add up fast.

Making Ethernet lossless (or close enough)

So the Ethernet crowd had to teach a lossy network to behave like a lossless one. That’s where a couple of acronyms come in, and if you’ve ever done serious QoS work for voice, some of this will feel familiar.

PFC (Priority Flow Control) is the IEEE 802.1Qbb mechanism that lets a switch tell its neighbor “pause this traffic class, I’m full,” instead of dropping packets. It works per priority, so you can make your RDMA class lossless while other traffic behaves normally.

The problem is that pauses propagate. One congested port pauses its upstream neighbor, which fills up and pauses its neighbor, and so on. Microsoft’s 2015 SIGCOMM paper that introduced DCQCN called out the issue directly: RoCEv2 “relies on Priority-based Flow Control (PFC) to enable a drop-free network,” but PFC “can lead to poor application performance due to problems like head-of-line blocking and unfairness.” (Source: Congestion Control for Large-Scale RDMA Deployments.)

ECN and DCQCN are the fix for that. Instead of slamming the brakes hop by hop, switches mark packets with ECN when queues start building, the receiving NIC sends a congestion notification back to the sender, and the sender eases off. Think of PFC as the emergency brake and DCQCN as easing off the gas before you get there.

Here’s the part I love, because it’s so real-world. Meta tried DCQCN, found it hard to tune for AI collectives, and then a firmware change introduced bugs that made it worse. For their 400G deployments, they turned DCQCN off entirely and ran with PFC plus congestion controls built into their collective library, reporting over a year of stable performance that way. They also leaned on deep-buffer switches in the spine to soak up the bursts.

Wait. What? The textbook answer got turned off in production?

Yes. And that’s the lesson I’d want every network engineer to take from this: the “right” design depends on your traffic, your hardware and your firmware, not on what the slide deck says. Meta also built a separate InfiniBand cluster alongside their RoCE one, both with 400 Gbps endpoints, and said they tuned both to equivalent performance for large training jobs. So the “RoCE vs. InfiniBand” argument is less of a cage match than the internet makes it sound. Both can work. Both take real engineering.

Your ECMP hashing is not ready for this

Most of us rely on ECMP (Equal-Cost Multi-Path) to spread traffic across links. It hashes on the five-tuple (source and destination IP, ports, protocol) and picks a path. With thousands of ordinary flows, the averages work out.

AI training doesn’t give you thousands of normal flows. It gives you a relatively small number of enormous ones, what Meta calls “fat network flows,” and their paper says the traffic has “low entropy in the UDP 5-tuple.” In plain English, the hash doesn’t have enough variety to work with, so a few links get hammered while others loaf around. Their fix was twofold: the collective library opens 16 flows between each pair of GPUs instead of one, and an enhanced ECMP scheme hashes on extra fields in the RoCE header to spread them out.

That’s not a GPU fix. That’s a networking fix, done by people who understand hashing, paths and polarization.

Rails, leafs and spines

Physical design matters just as much. If you’ve built a leaf-spine (Clos) fabric for a data center, you already know the basic shape. AI clusters take it further.

Meta’s RoCE cluster for Llama 3, as described in the paper, is a three-layer Clos. Each rack holds 16 GPUs across two servers behind one top-of-rack switch. 192 of those racks form a pod of 3,072 GPUs with full bisection bandwidth, meaning no oversubscription. Then eight pods connect at the aggregation layer into a 24,000-GPU cluster, and that top layer is oversubscribed at 1:7. That’s why their job scheduler and parallelism are topology-aware; they try hard to keep the chattiest traffic inside a pod.

NVIDIA’s reference designs add another twist called rail optimization. In the DGX SuperPOD H100 reference architecture, each server has eight compute NICs, one per GPU, and the same-numbered NIC on every server plugs into the same leaf switch. GPU 1 on every box shares a “rail,” GPU 2 on every box shares another, and so on. NVIDIA says traffic on a rail is always one hop away from the other 31 nodes in its scalable unit; only traffic between rails or between units crosses the spine. Since collective libraries tend to talk rail to rail, most of the heavy traffic stays one hop away.

It’s a little like assigning every lane on the interstate to a specific exit, so nobody has to merge. Weird the first time you see it. Brilliant once it clicks.

Scale-up vs. scale-out

Scale-up is the network inside the box (or, lately, inside the rack). That’s NVIDIA’s NVLink. According to NVIDIA’s NVLink specifications, fifth-generation NVLink on Blackwell gives each GPU 1,800 GB/s of bandwidth, and a GB200 NVL72 rack connects 72 GPUs at 130 TB/s total. NVIDIA lists sixth-generation NVLink for its upcoming Vera Rubin platform at 3,000 GB/s per GPU (preliminary specs, subject to change).

Scale-out is the network between those boxes and racks: the InfiniBand or Ethernet fabric we’ve been talking about, typically at 400 Gb/s per GPU in the Meta clusters above.

Put those side by side and the gap is wild. NVLink’s figures are bidirectional, so the fair comparison is a 400 Gb/s NIC in both directions, which is about 100 GB/s. NVLink 5 is roughly 18 times that per GPU. That’s why engineers try to keep the most talkative kind of parallelism (tensor parallelism) inside the NVLink domain, and push the less chatty kinds out across the fabric. In Llama 3’s configuration, tensor parallelism stayed at 8, the same as the number of GPUs connected by NVLink in each server. Coincidence? Of course not.

It’s the same idea as keeping your heaviest traffic on the backplane instead of the uplinks. You’ve done this before. The numbers just have more zeros.

Ultra Ethernet: Ethernet wants the whole cake

InfiniBand has been the default for high-end AI training for a long time, but the industry clearly wants an open, multi-vendor Ethernet option built for this work instead of bolted on. Enter the Ultra Ethernet Consortium, which operates under the Linux Foundation.

On June 11, 2025, UEC released Specification 1.0, covering NICs, switches, optics and cables. According to the UEC 1.0 white paper (written by Intersect360 Research and sponsored by UEC, so season to taste), the spec brings a modern RDMA transport for Ethernet, sender-based congestion control designed for microsecond-level latency, multipath congestion control with coordinated packet spraying, and optional receiver-side credits to deal with incast. Meta, Microsoft and Oracle are listed as steering members.

Some of that should sound familiar by now. Packet spraying goes after the same problem as Meta’s 16 flows, and receiver credits rhyme with InfiniBand’s approach. Hard-won production pain, written into a standard.

Will it win? I don’t know, and I’d be suspicious of anybody who claims they do. Engineers who understand why these features exist will be ready either way.

It’s not just training: inference needs the network too

Inference (actually running the model to answer prompts) has its own networking story. One big technique right now is disaggregated serving: splitting the “read the prompt” phase (prefill) and the “write the answer” phase (decode) onto different GPUs. The prefill side builds a chunk of state called the KV cache, then ships it to the decode side. NVIDIA’s Dynamo documentation calls that KV transfer “the critical path” and says production cross-node deployments usually need RDMA. Their Kubernetes communication guide reports observing roughly 98 seconds to first token over TCP versus about 200 to 500 milliseconds with RDMA.

Ninety-eight seconds! You could brew a pot of coffee.

Then there’s the other end of the spectrum: small models at the edge. In my post on AI inference rationing, I argued that a lot of everyday AI will end up on laptops, phones and boxes in the wiring closet. And if you’re weighing whether to run that stuff yourself, my rent vs. own vs. hybrid TCO model is a good place to plug in your numbers. Edge inference doesn’t need a 24,000-GPU fabric, but it does need someone thinking about segmentation, QoS, bandwidth to the model server and what happens when the WAN hiccups. Sound like anyone you know?

What this means for us enterprise network engineers

Let me be clear: most of us aren’t going to build a 24,000-GPU cluster. That’s hyperscaler territory. But the skills behind those clusters are leaking into the enterprise, and I think that’s great news for the folks who turn the wrenches.

Years ago, a Cisco Press QoS book by Michael Cavanaugh and Wendell Odom made “Cav” what I call an unaware mentor; he had no idea he was teaching me. Queuing, drops and priority all show up again here. Lossless classes, buffer thresholds, marking, head-of-line blocking, jitter. Voice people have been sweating packet loss and latency forever. AI just raised the stakes.

If I were sharpening a toolkit for this, here’s where I’d spend time:

• RDMA fundamentals. Understand verbs, queue pairs and why RDMA hates loss.
• Data center QoS done right. PFC, ECN, DSCP marking and buffer tuning.
• Clos and leaf-spine design. Oversubscription math, ECMP behavior with low-entropy flows, and rail-optimized layouts.
• Telemetry. Pause frame counters, ECN marks, queue depth, per-link utilization. In a “functioning but slow” world, you can’t fix what you can’t measure.
• Speaking the AI team’s language. Learn enough about all-reduce, NCCL and parallelism to have a real conversation. Running nccl-tests and understanding the output goes a long way.
• Automation. Fabrics at this scale aren’t configured by hand. Python and a source of truth aren’t optional anymore.

I don’t have a degree in any of this. I figured it out, and you can too. Nobody is going to hand you an AI networking certification that makes you an expert overnight, but the fundamentals you already have (queuing, routing, hashing, troubleshooting under pressure) are exactly what these clusters need.

The GPUs do the thinking. The network decides whether they get to.

Pick one thing from the list above and lab it this month. Then come tell me about it in the comments or find me on LinkedIn. I’m just here to ask the tough questions.

Sources and further reading

• Meta, “The Llama 3 Herd of Models”, 2024 (infrastructure, network topology and reliability sections).
• Adithya Gangidi et al., “RDMA over Ethernet for Distributed AI Training at Meta Scale”, ACM SIGCOMM 2024.
• Meta Engineering, “Building Meta’s GenAI Infrastructure”, March 2024.
• Yibo Zhu et al., “Congestion Control for Large-Scale RDMA Deployments”, ACM SIGCOMM 2015 (DCQCN).
• Jeffrey Dean and Luiz André Barroso, “The Tail at Scale”, Communications of the ACM, 2013.
• NVIDIA, nccl-tests performance documentation (all-reduce bandwidth math).
• NVIDIA, NVLink and NVLink Switch specifications.
• NVIDIA, DGX SuperPOD H100 reference architecture: network fabrics.
• NVIDIA, Dynamo disaggregated serving documentation.
• Ultra Ethernet Consortium, UEC Specification 1.0 announcement and white paper, June 2025.

Keep reading

If this got your gears turning, here’s what I’d read next:

• The Machines Were Always Trying to Talk. The Hardware Finally Let Them. How AI works under the hood, from tokens to transformers, and why hardware made it possible.
• AI Inference Rationing: Is This the Day the Music Died? What happens when the world wants more AI than it can power.
• AI Rent vs. Own vs. Hybrid: A Free TCO Decision Model in Excel. Run your own numbers on renting vs. buying AI capacity.