SD-WAN Underlay vs Overlay: How to Tell Which Side Is Broken

Picture the ticket. Site A can reach Site B on the dashboard. The tunnel icon is green. Somebody in finance still cannot open the ERP. Or the flip side: the tunnel is red, everyone is yelling about SD-WAN, and a traceroute from the WAN interface never leaves the ISP’s first hop. Same symptom family. Different broken thing.

I have spent a lot of years living in that gap, building and running SD-WAN in production (I also wrote about keeping networks apart on purpose in IT/OT network segmentation). The gear brands change. The question does not: which side is broken?

This is my practical split between underlay and overlay troubleshooting. Vendor-neutral enough to use on Monday. Specific enough that I cite the docs when I name a knob. No magic. Just a way to stop chasing the wrong light.

Underlay and overlay in plain English

Think of the underlay as the road and the overlay as the convoy. The road is whatever the carrier gave you: Internet DIA, MPLS, LTE, a metro Ethernet handoff. If the road is flooded, the convoy does not matter. The overlay is the encrypted tunnels, the control sessions, the probe traffic and the business policy that decide which truck takes which lane. If the road is fine and the convoy is lost, you have an overlay problem.

Cisco’s Catalyst SD-WAN design guide puts it the same way without the truck metaphor: the underlay is the physical infrastructure and transport network, and the overlay is the fabric you build on top of it. On that platform, WAN interfaces live in the transport VPN (VPN 0). That is where underlay routing has to work before anything fancy happens. See the Cisco Catalyst SD-WAN Design Guide if you want the official wording.

Other vendors use different nouns for the same idea. Broadcom’s VMware/VeloCloud edge talks about paths and VCMP over UDP 2426. Fortinet talks about SD-WAN members and performance SLAs. The labels change. The stack does not: carrier path first, tunnel and probes second, application third.

Control plane vs data plane (and why both can lie)

Here is where people get burned. SD-WAN has a control plane (talking to controllers/orchestrators, learning remote sites, distributing keys and policy) and a data plane (actually forwarding user packets across tunnels). Those two can disagree, loudly.

On Cisco Catalyst SD-WAN, BFD runs on the IPsec (or GRE) data-plane connections between WAN Edges. It is separate from the DTLS/TLS control connections to the controllers. Cisco’s BFD monitoring guide is blunt about the dashboard math:

• BFD 0/0: control connection to the SD-WAN Controller is down. Total sessions and up sessions both go to zero.
• BFD 0/1: control is still up, but an underlay or IPsec issue knocked the data-plane session down. You expected a session; it is not up.

That 0/1 case is the classic “controller looks happy, users are not” ticket. Cisco documents it in the BFD protocol monitoring guide (26.x). When BFD drops, the route pointing over that tunnel is removed from the FIB even if it still sits in the RIB. In human terms: the box still knows the route, but it will not use that next hop until BFD recovers.

VMware/VeloCloud tells a similar story with different words. The Administration Guide’s link monitoring colors are Stable (stats arriving consistently), Unstable (expected stats missing, or the link is down but has not been inactive for 10 minutes yet), and Disconnected (inactive more than 10 minutes). The same guide documents VCMP, the multipath encapsulation, on UDP 2426. If that UDP path is filtered while ICMP still works, you will chase ghosts. See the VMware SD-WAN Administration Guide 4.3 (PDF).

TLOCs, colors and “the tunnel” people point at

If you work Cisco Catalyst SD-WAN, a TLOC (transport locator) is how the fabric names a WAN attachment. Cisco defines it as the combination of system IP, color and encapsulation (IPsec or GRE). Color is not paint; it is the label for the transport type (mpls, biz-internet, lte, public-internet, and so on), and it matters for how public vs private addressing is treated. Official definition: TLOC in Catalyst SD-WAN.

Why care during an outage? Because “the tunnel is down” is not a diagnosis. Which TLOC pair? Which color? Is BFD down on mpls but up on biz-internet while policy still prefers mpls for that app? You can have one transport healthy, one sick, and a business policy that insists on the sick one. That is not an underlay failure at the site. That is an overlay choice meeting a degraded path.

Non-Cisco platforms have equivalents: a WAN link plus a path to a peer, or an IPsec phase-2 plus a health-check member. Whatever your GUI calls it, name the exact path before you open a carrier ticket.

“Tunnel up” can still be a blackhole

Status lights measure what they were told to measure. They do not measure your ERP login.

A few ways the light stays green while the app dies:

• Underlay ICMP works; overlay UDP/ESP does not. Intermediate firewalls and carrier scrubbers love to treat probe traffic differently from encrypted tunnel traffic. On VeloCloud that often means UDP 2426 (VCMP) is treated worse than ICMP; on IPsec fabrics it is ESP or UDP 4500. If your proof is only ping to the far WAN IP, you have not proved the overlay.
• BFD/SLA uses a DSCP the carrier treats like royalty; your app does not. Cisco notes BFD control/echo often marked DSCP 48. If the ISP protects DSCP 48 and left-turns your DSCP 46 voice into a ditch, probes look fine and calls sound like a blender.
• Return path is broken. One-way underlay, asymmetric routing, or a stateful firewall that never saw the forward flow. Same family of pain as SIP one-way audio: signaling (or tunnel setup) succeeded, media (or app data) did not.
• Policy steered you into a dead end. App-aware routing, an SD-WAN rule, or a “preferred color” sent the flow to a member that is up in the abstract and useless for this destination.
• You are testing the wrong VRF/VPN. Management path works. Service VPN does not. Congrats on pinging yourself.

Brownout vs hard fail

Hard fail is easy emotionally. Link down. BGP neighbor gone. BFD dead. Carrier NOC has a ticket number. Brownout is the mean cousin: the circuit is “up,” average latency looks tolerable on a five-minute chart, and voice, VDI and chatty SaaS are already on fire.

This is why SD-WAN products obsess over loss, latency and jitter on the overlay, not just link state. Cisco’s application-aware routing uses BFD hellos on data-plane tunnels to measure those three, then classifies tunnels into SLA classes. Defaults you will see in the Application-Aware Routing guide include a 1-second BFD hello, multiplier 7 for liveliness, a 10-minute app-route poll interval, and a sliding window of six poll buckets for SLA decisions. That damping keeps you from flapping every time a truck rolls past the cell site. It also means a short brownout can hurt users before policy finishes arguing with itself. Tune timers on purpose; do not leave them on folklore.

Fortinet’s performance SLAs play the same game with health-check probes against a server you choose. Their diagnose commands (SD-WAN diagnose commands) show member alive/dead plus loss, latency and jitter. Community guidance on flapping SLAs is worth a read too: aggressive failtime plus update-static-route can withdraw routes for everyone when a probe burps (health-check flapping tip). A flapping SLA is an overlay decision problem that often has an underlay root cause, or a bad probe target.

Rule of thumb I use: if users feel pain and every tunnel still says up, stop staring at link state. Pull loss/jitter for the exact path the app is using. Brownout lives in the histograms.

A diagnosis order that actually finishes tickets

Hope is not a strategy, and neither is restarting the edge “just to see.” Work the stack top-down only after you have proved the bottom.

1. Freeze the symptom

• Which sites, which application, which direction, since when?
• Is it all apps or one app? All transports or one color/member?
• Did a change land (policy, template, ISP maintenance, firewall rule)?

2. Prove underlay reachability for the suspect transport

• Interface up/up, IP, gateway, DHCP or static as expected.
• Ping and traceroute from the WAN VRF/VPN 0 to the far WAN IP (and to a known Internet target if it is DIA).
• Check ISP handoff errors, optical levels, LTE RSSI/RSRP, shaping, and whether BGP (if used) is established with expected prefixes.
• Confirm nobody filtered ESP, UDP 500/4500, UDP 2426, or whatever your overlay needs. Underlay “ping works” is necessary and not sufficient.

3. Prove overlay liveliness on that same transport

• Cisco: BFD sessions for the TLOC pair, flap history, IPsec state. Tunnel Troubleshooting in newer Manager releases walks TLOC, IPsec, SD-WAN session, NAT and BFD (BFD tunnel troubleshooting). Also confirm allow-service bfd (and friends) on the tunnel interface when transport-side filters are in play (common BFD errors).
• VeloCloud: List Paths, path state, loss/latency/jitter, VPN state.
• Fortinet: diagnose sys sdwan member, diagnose sys sdwan health-check status, and the service rule that should have selected a member.

4. Prove the app path

• Which member/color did policy pick? Is that member in SLA?
• Flow logs / DPI / SAIE: did the session form? Reset? Stall after three-way handshake?
• Test from a client in the correct service VPN, not from the router management plane.
• Compare overlay ping/traceroute (when the platform offers it) with a real transaction. They are related; they are not the same.

5. Only then escalate

Carrier ticket with underlay evidence. Vendor TAC with overlay evidence. App owner with flow evidence. Send each party the layer they own. Mixing them is how a four-hour outage becomes a four-day conference call.

What to capture (so you are not arguing from vibes)

Minimum kit when the ticket is hot:

• Timestamps in one timezone. User report, edge event log, controller event, ISP NOC. Align them or go home.
• Underlay: interface stats, traceroute, ICMP and (when safe) a short iperf or OWAMP-style test on the transport. Note DSCP if you can set it.
• Overlay: BFD/SLA/path tables before and after; IPsec SA state; packet captures filtered to tunnel ports between peers when the platform says loss and underlay ping does not.
• Policy: the exact app-route / SD-WAN rule / ACL that matched, and the member it selected.
• App: client and server captures if you can, or at least firewall allow/deny hits and retransmit counters.

If you only have time for one clever trick, do this: underlay ping to the peer WAN IP, overlay probe stats for that same pair, and one real application attempt, all inside the same five-minute window. Three rows. One story.

Common false positives (ask me how I know the feelings, not the fictional war stories)

• “Internet is down” because DNS is down. Underlay is fine. Overlay is fine. Resolver path or DNS policy is not.
• “SD-WAN broke voice” when SIP ALG or NAT ate RTP. Different layer. Start with one-way audio troubleshooting before you rearrange TLOC preference.
• Probe target is bad. Public DNS that rate-limits ICMP, a server that is itself sick, or a health-check sourced from the wrong interface. The SLA is lying because you taught it the wrong exam.
• Dashboard delay. The outage ended. The graph has not. SSH to the edge before you declare victory or defeat.
• Secondary transport looks “unused” and therefore “broken.” Idle paths go quiet. Quiet is not dead. Force traffic or read the probe table.
• OT vs IT path confusion. In segmented shops, the SCADA path is not the office path even when they share a campus. If you care about that boundary, read the Purdue / IEC 62443 piece and do not troubleshoot across a conduit you meant to keep closed.

App-aware routing and business policy: when the overlay chooses poorly

Sometimes every transport is “up” and the fabric still makes a dumb choice for a specific application. That is the point of app-aware routing and SD-WAN rules: steer by loss/latency/jitter and by business intent. It is also how you get a self-inflicted brownout.

On Catalyst SD-WAN, an SLA class sets maximum loss, latency and jitter. Traffic matching an app-route sequence is sent on tunnels that meet that class, with options for preferred color, strict drop, backup color and (in later releases) fallback to best path. If you set strict and nothing meets SLA, you chose an outage over a degraded path. That can be correct for a regulatory flow. It is surprising for email. Read the policy before you blame the carrier.

Fortinet service rules and VeloCloud business policies rhyme. Match the flow, pick a strategy (best quality, meet SLA, lowest cost), pick members. When the ticket is “Teams is bad but the tunnel is up,” dump the rule that matched Teams and the member it picked. Half the time the underlay is only slightly soggy and the policy is all-or-nothing.

And yes, agentic AI tooling can help you gather show-command output faster. It cannot tell you which layer owns the failure until you teach it this split. (I wrote about where that helps and where it does not in AI agents and network engineering.) Tools accelerate. Fundamentals decide.

A short field checklist

1. Name the path: site pair, transport/color/member, VPN/VRF, application.
2. Underlay proof: L1/L2, IP, traceroute, carrier errors, required UDP/ESP allowed.
3. Overlay proof: BFD/path/SLA state for that exact pair, not a different color.
4. Policy proof: which rule selected which member, and did that member meet SLA?
5. App proof: real client flow in the right segment, not a management ping.
6. Brownout check: loss/jitter histograms, not only up/down.
7. One change at a time. Write it down. Your future self is also on-call.

Closing

SD-WAN did not invent underlay versus overlay. It just made the overlay loud enough that people forget the underlay still has a veto. When the tunnel is up and the app is dead, believe the app, then interrogate the path with three separate witnesses: carrier reachability, tunnel probes, and the flow itself.

Split the stack. Fix the right side. Ship the evidence with the ticket.

And when one of these tickets turns into a real outage, the conversation the morning after matters as much as the fix. I wrote up how I’d run blameless postmortems for network teams so the next one gets reported fast instead of hidden.

If this saved you an hour of mutual blame between NetOps and the ISP, send it to the group chat before the next maintenance window. And if you are the ham in the NOC eating cold coffee at 2 a.m., 73. We have all been there.

The short version

When the tunnel looks up and the app is still dead, odds are you’re looking at the wrong layer. The underlay is the ISP, MPLS or LTE path that carries the packets, and the overlay is the encrypted fabric, the probes and the policy riding on top of it. The control plane can look perfectly healthy while the data plane is not. So prove underlay reachability first, then overlay BFD and SLA, and only then go after the app path.

References

• Cisco Catalyst SD-WAN Design Guide (underlay vs overlay, VPN 0, TLOC/color).
• Cisco Catalyst SD-WAN: TLOC
• Cisco Catalyst SD-WAN: BFD protocol monitoring (26.x) (including 0/0 vs 0/1).
• Cisco: Troubleshoot common BFD errors
• Cisco: Troubleshoot BFD tunnel issues
• Cisco: Application-Aware Routing (SLA, BFD timers, brownout handling)
• VMware SD-WAN Administration Guide 4.3 (PDF) (link Stable/Unstable/Disconnected; VCMP UDP 2426)
• Fortinet: SD-WAN diagnose commands
• Fortinet community: health-check flapping and route churn