using WordPress and 
Imagine being alone in a small, dimly lit IDF closet on the third floor of a hospital at midnight, and the switch you were just working on is no longer passing traffic. I don’t have to imagine it. That was me, early in my career, and I caused it. Within minutes, fives of people seemed to have spawned behind me like characters in a dystopian role-playing video game, each one trying to fix a problem I made.
I wrote about that night in my book, and the part that stuck with me wasn’t the switch. It was the conversation afterward with Brian, one of my earliest mentors, who was honest enough to tell me the real problem was my arrogance. He was right.
Here’s the thing I’ve come to believe after 20-plus years of turning wrenches in networks: how a team talks about an outage the morning after decides whether the next one gets reported in five minutes or hidden for five hours. That morning-after conversation has a name. It’s the blameless postmortem, and most network teams either don’t do them or do a version that’s really just a polite firing squad.
So let’s fix that. This is how I’d run a blameless postmortem for a network team, from lowercase point a, to UPPERCASE POINT B, with the network-specific stuff the software-flavored guides tend to skip.
What a blameless postmortem actually is (and isn’t)
Google’s SRE book defines a postmortem as “a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent the incident from recurring.” The blameless part means the write-up focuses on contributing causes “without indicting any individual or team for bad or inappropriate behavior,” and it assumes everyone “had good intentions and did the right thing with the information they had.” Don’t take my word for it; go read the Postmortem Culture chapter yourself. It’s short. I’ll wait.
The same chapter points out that blameless culture came out of healthcare and avionics, industries “where mistakes can be fatal.” That should land with anybody who works in critical infrastructure, because our mistakes don’t stay on a dashboard. They show up at a nurses’ station or in a control room.
Now, one more time for the folks in the back: blameless does not mean consequence-free, and it doesn’t mean nobody’s name appears in the document. Atlassian says it plainly in their blameless postmortem guide: the point isn’t that you never identify who made a mistake; the point is to “remove the fear of looking stupid, being reprimanded, or even losing your job” so people tell the truth. Somebody still owns every action item. Somebody still has to change how they work. The difference is that we’re asking what made the mistake easy, not who we get to be mad at.
Google has a line in there I love: “You can’t ‘fix’ people, but you can fix systems and processes.” Write that on the whiteboard in the NOC.
Why network teams are worse at this than software teams
I’m going to be a little contrarian here. I think network folks have a harder time with blameless culture than developers do, and it isn’t because we’re meaner (well … mostly).
Software teams usually have version control, CI pipelines and deploy logs baked into how they work, so the “what changed” question answers itself. A lot of network changes still happen by a human at a CLI prompt, in a maintenance window, often alone, sometimes at midnight in a closet. When something breaks, there’s exactly one set of fingerprints on the keyboard. That makes the blame reflex almost automatic, because the person is the most visible part of the system.
We also tend to carry a certain hero culture. The engineer who can fix anything at 2 a.m. gets the street cred, and the engineer who caused the 2 a.m. call gets the side-eye. Both of those reactions skip right past the useful questions: why was a change that risky possible with one command, why did monitoring take twenty minutes to notice, and why was the rollback plan “I’ll figure it out”?
That’s the hospital closet in a nutshell. I was the most visible part of the system, so I was the obvious answer. Brian’s feedback on my arrogance was fair, and it changed how I work. But a good postmortem would have asked a second question too: what about that setup let one overconfident kid take down a hospital floor at midnight?
Both questions matter. Most teams only ask the first one.
Decide what triggers a postmortem before you need one
If you only hold postmortems for the outages that made the CIO’s phone ring, people learn that a postmortem is a punishment for getting caught. Google’s advice is to define the criteria ahead of time so everyone knows when one is coming. Their common triggers include user-visible downtime beyond a threshold, any data loss, on-call intervention like a rollback or rerouting traffic, resolution time over a threshold, and a monitoring failure where a human found the problem before the tools did.
For a network team, I’d translate that into something like this, and I’d write it down where everyone can see it:
- Any unplanned loss of connectivity for a site, a clinical or operational system, or a voice platform beyond a set number of minutes (pick the number with your business owners, not by yourself).
- Any change that had to be rolled back, even if users never noticed.
- Any outage a user reported before our monitoring did.
- Any security-relevant change that went sideways, like an ACL or firewall rule that opened or closed the wrong thing.
- Anything an engineer asks for. Anyone can call one, and calling one is never held against them.
That second-to-last bullet is where IT/OT folks should pay attention. If you’ve drawn real zones and conduits (I wrote a whole post on IT/OT network segmentation with the Purdue model and IEC 62443), then a conduit rule that quietly changed is a postmortem, full stop, even if nothing fell over.
Build the timeline first, and build it forward
Before anybody talks about causes, get the timeline right. Atlassian calls this out because a misunderstanding of what actually happened will send the whole meeting off the rails, and they’re correct. In network land, the timeline is where most of the real truth lives.
Here’s what I’d pull, roughly in this order:
Change records. The ticket, the approved plan, the maintenance window, and, more importantly, what was actually typed. If your gear keeps a config archive or commit history, grab the before and after diff.
Syslog and SNMP traps from every device in the blast radius, not just the one that was touched. Interface flaps, spanning tree topology changes, routing adjacency drops and power events tell the story in order. (Yes, this only works if NTP is configured and every box agrees what time it is. If your timestamps disagree, that’s your first action item.)
Monitoring and alerting history. When did the first alert fire, who got it, and when did a human acknowledge it? The gap between “it broke” and “we knew” is often the most useful number in the whole document.
Chat and bridge logs. Who joined and what they tried, in order. This is where you’ll find the three people who all rebooted the same thing.
The user side. Tickets and calls from the people who were affected. For a voice outage, that might be carrier tickets and call detail records; I covered how messy that evidence gets in SIP one-way audio troubleshooting.
Then build the timeline forward. PagerDuty’s blameless postmortem guide has a smart trick for fighting hindsight bias: start the timeline at a point before the incident and work forward, instead of starting at the resolution and working backward. When you work backward, every decision looks obviously dumb, because you already know how the movie ends. When you work forward, you see what the engineer saw at the time, which was a lot less.
This is also a good spot for agentic AI, if your team uses it. Having an agent stitch syslog, alerts and chat into a first-draft timeline saves real hours. Just treat that draft the way you’d treat a junior engineer’s notes: useful, fast, and checked by a human before anyone makes a decision on it. (More on how I think about agents in AI agents in network engineering.)
Run the meeting like a debug session, not a deposition
The meeting is where blameless culture lives or dies. You can have the prettiest template in the world, and one manager saying “so who approved this?” in the wrong tone will set you back six months.
A few rules I’d hold the line on:
Ask what and how, not who and why
PagerDuty’s guide recommends “what” and “how” questions over “who” and “why,” and it explains the reasoning well: “why” questions force people to justify themselves, which is blame wearing a polite hat. Compare these two:
- “Why did you shut down that uplink?”
- “What did the interface counters show you right before you shut down that uplink, and how did you decide it was the bad link?”
Same information. Completely different temperature in the room. The second one gets you the actual decision path, which is the part you can fix.
Talk about an unnamed engineer
The same guide suggests abstracting to an unnamed responder and asking why a reasonable, rational and decent person might have taken that action. That’s a great habit for network teams. “An engineer looking at this topology diagram would reasonably assume port 48 was the uplink” points straight at the stale diagram. “Dave assumed port 48 was the uplink” points straight at Dave.
Have the person closest to the change tell the story first
Let them walk the room through it while the details are fresh. If the culture is right, they’ll volunteer the embarrassing parts on their own, and those are almost always the most valuable parts. If they won’t, that tells you something about your culture, not about them.
Leaders talk last
Leadership is not glorious. If you’re the manager, your job in this meeting is mostly to protect the conversation and keep it moving, not to deliver the verdict. The minute the senior person states a conclusion, everyone else stops thinking out loud.
Finding causes without playing the “five whys” lottery
Plenty of teams run a “five whys” exercise and stop when they hit a person. “Why did the core go down? Because the engineer pushed a bad config. Why? Because he fat-fingered the VLAN.” Done, root cause found, case closed, engineer feels terrible.
That isn’t a root cause. That’s a suspect.
Complex systems rarely fail for one reason, so I’d rather list contributing factors and sort them into buckets. For network incidents, these are the buckets I keep coming back to:
[The change itself] Was it peer reviewed? Was there a tested rollback? Was it the kind of change your gear can protect you from? Junos, for example, has commit confirmed, which automatically rolls back the configuration after 10 minutes by default unless you confirm it. On classic Cisco IOS, the old-school version of that safety net is scheduling a “reload in” before a risky remote change and canceling it once you’re sure you didn’t just cut off your own access. If nobody on the team uses those tools, that’s a training and standards finding, not a character flaw.
[The information the engineer had] Were the diagrams current? Did the port descriptions match reality? Did the runbook exist, and did it say what the engineer thought it said?
[Detection] How long until monitoring noticed, and did the alert go to someone awake?
[Response] Did the bridge have a clear incident lead, or did fives of people all start fixing things at once? (Ask me how I know.)
[Conditions] Was this done at midnight after a twelve-hour day, solo, in a closet with no second set of eyes? Fatigue and isolation are system conditions. You scheduled them.
Atlassian shares a story in that same guide that every network engineer should read. An engineer made a syntax mistake in a config file for a piece of critical equipment, and it took the entire company down for 45 minutes. Instead of shaming the engineer, they ran a blameless postmortem, and the permanent fix was an automated “will it start” check on the config before loading, and eventually removing human hands from that system’s configuration altogether. The engineer kept their job and, according to Atlassian, still adds a lot of value. That’s the whole philosophy in one paragraph: the human error was real, and the fix was in the system.
A postmortem template a network team will actually fill out
Google publishes a full example postmortem if you want the gold-standard version. For most network teams, I’d start leaner. If the template takes longer to fill out than the outage lasted, nobody will use it.
- Summary: two or three sentences a non-engineer can read.
- Impact: which sites, systems and users, for how long, and how we measured it.
- Timeline: built forward, with timestamps and sources (syslog, alerts, chat, tickets).
- Detection: how we found out, and how long that took.
- Contributing factors: sorted into the buckets above, written about the system, not the person.
- What went well: do not skip this. Somebody probably made a smart call under pressure.
- Action items: each with one owner, a due date and a ticket number.
- Evidence: config diffs, captures, graphs, links.
That “what went well” section isn’t fluff, by the way. PagerDuty’s guide talks about negativity bias, our tendency to magnify the bad stuff, and recommends describing what was handled well so the team doesn’t walk away demoralized. Google tells a story about an SRE whose change took down a critical service for four minutes; because he rolled it back right away, he got peer bonuses and a round of applause at an all-hands. Four minutes was short because he moved fast. That’s the behavior you want copied.
Action items are where postmortems go to die
You know the check-engine light that’s been on in somebody’s car for two years? They “know about it.” They’re going to “get to it.” That’s what most postmortem action items look like six months later.
A few habits help:
Keep the list short and real. Three action items that get done beat twelve that live forever in a spreadsheet. Prefer fixes that change the system (an automated pre-check, a config template, a monitoring rule, a peer-review requirement for core changes) over fixes that ask humans to try harder. “Be more careful” is not an action item. It’s a wish.
Then review them. Google’s line is that “an unreviewed postmortem might as well never have existed,” and I’d extend that to action items. Put open postmortem items on the agenda of your regular team meeting until they’re closed or someone makes a conscious, written decision not to do them. Either is fine. Silence isn’t.
And when an action item turns into a network-wide standard, write it up somewhere people will find it again. If your team has a habit of chasing the wrong layer, for example, a shared troubleshooting split like the one in SD-WAN underlay vs overlay troubleshooting is the kind of thing that started life as a lesson learned and earned its way into the runbook.
Selling it upstairs (and to the team)
Here’s where I put my leadership hat on, because the hardest part of blameless postmortems isn’t the template. It’s trust.
In my book I wrote that trust is given on the back of vulnerability; if trust is the car, then vulnerability is the road. A blameless postmortem asks an engineer to be vulnerable in front of their peers and their boss. They’ll only do that if they believe it’s safe, and they’ll only believe it after they’ve watched it be safe a few times. Atlassian makes the same point from the other direction: if one postmortem is blameless and the next one isn’t, the whole thing falls apart. Consistency is the product.
For executives, I’d skip the philosophy and talk risk. PagerDuty’s guide notes that blame makes engineers hesitate to speak up, which increases time to acknowledge and time to resolve. That’s language a CIO understands. Every minute an engineer spends deciding whether it’s safe to admit they touched something is a minute the outage keeps running. In critical infrastructure, those minutes matter to real people.
For the team, go first. If you’re the lead, write up one of your own mistakes using the new template and walk the team through it. Nothing says “this is safe” like the boss volunteering to be the cautionary tale. I’ve been the cautionary tale in a hospital closet; it didn’t kill me, and it made me a better engineer. Brian could have just chewed me out. He gave me honest feedback instead, and I’ve tried to pay that forward ever since.
Leadership, at least the kind I believe in, is in the leader-making business. Blameless postmortems are one of the cheapest leader-making tools you’ll ever find.
Your first one this month
Don’t wait for the big one. Pick a recent small incident, maybe a rollback nobody outside the team noticed, and run it through this process with a 45-minute meeting and the lean template. Build the timeline forward. Ask what and how. End with three action items that each have one owner. Then do it again next month, and keep doing it until it stops feeling weird.
Like the man said in that Whitesnake song, here I go again. Every outage is a rerun unless you learn something from it. Hope isn’t a strategy, and neither is hoping nobody touches that switch again.
If you’re building this out on your team and want to compare notes, I’d love to hear what’s worked for you and what blew up in your face. Figure it out together. It’s faster that way.
73,
Ryan
No responses yet