Case guides · Under pressure · UP-01

How do I lead a major incident as the manager, not the fixer?

Split the roles out loud in the first five minutes: a commander who coordinates and doesn't fix, an operations lead who changes production, and a communications lead who updates everyone else. Every change is announced before it's made. Mitigate first with the safest reversible step, and understand later.

The situation

It's 9:40 on a weekday and checkout is failing for a third of your customers. The bridge call has fourteen people on it. Two of your strongest engineers are debugging in parallel and changing things without telling each other. A director has joined and is asking for an ETA every four minutes. You used to be the person who fixed this kind of thing, and every instinct you have says: open a terminal. Your job right now is the opposite. Somebody has to run the room, and if it isn't you, nobody will.

EM Field Guide is practice. It is not legal, HR or medical advice. If a situation involves dismissal, a performance plan, discrimination, safety or health, talk to your HR team or a qualified professional.

What matters most

  1. Split the roles out loud, in the first five minutes. Google's SRE book (chapter 14, "Managing Incidents") describes roles borrowed from the emergency services' Incident Command System, including an incident commander who holds the overall picture and assigns work, an operations lead who is the only one changing production, and a communications lead who handles updates to everyone else (a fourth, planning, supports longer incidents). Say who is in each seat. If you don't delegate a role, you hold it by default.
  2. The commander doesn't fix. PagerDuty's public incident response guide is blunt about this: the incident commander coordinates and decides, and takes no remediation actions. The moment you start typing commands, nobody is watching the whole board. If you're the best person to fix it, hand the commander role to someone else, explicitly, and say so on the bridge.
  3. One set of hands on production. Parallel, unannounced changes are how a one-hour outage becomes a four-hour one. Every change goes through the operations lead, is announced before it's made, and is logged. "Before you run that, say it on the bridge."
  4. Protect the responders from the audience. Executives need information, not access to engineers. Give them a predictable update rhythm from the communications lead (for example every 30 minutes, even when the update is "no change") and a separate channel. PagerDuty's guide goes as far as saying the commander should be willing to remove anyone from the call who is distracting it, however senior.
  5. Mitigate first, understand later. Restore service with the safest reversible step (roll back, fail over, turn off the feature) before anyone finds the root cause. The postmortem is where you learn; the bridge is where you stop the bleeding.

How this lands in teams across Asia

The role model above assumes that the person in the commander seat can overrule anyone on the call. In many teams across the region, that assumption breaks the moment someone more senior joins.

  • Seniority silences the bridge. When a VP or a long-tenured architect joins, junior responders often stop reporting what they see and start waiting to be told. Nobody wants to contradict the most senior voice in front of others, even when they have the key log line. Agree before incidents, with your own leadership, that during an incident the commander's role outranks job title. Then, on the call, name it: "For the next hour I'm running this as incident commander. [Name], you're operations lead, you speak for the fix."
  • Ask by name, not "any questions?" An open "anyone seeing anything?" gets silence in a room where speaking up uninvited feels presumptuous. Ask specific people specific questions: "Priya, what does the payment queue look like right now?" You're giving permission as much as asking.
  • Face works against good news that's bad news. An engineer who suspects their own deploy caused it may not say so on a call with senior people present. Make the path private: "If anyone has a hunch about a recent change, message me directly, no need to say it here." Then act on it without naming them.
  • Multi-country bridges. When responders span several countries and time zones, the language of the bridge matters. Speak slowly, repeat decisions in writing in the incident channel, and don't treat silence from a second-language speaker as agreement. For incidents that run long, plan a hand-over to the next region's on-call before people are exhausted, not after.
  • The escalation reflex. In strongly hierarchical organisations, leaders may expect to be phoned personally rather than read a status page. Agree the update path in advance, or you'll spend the incident on the phone upward instead of running it.

A way to open

“I'm going to be incident commander.”

The rest of the script is in the app.

Rehearse it

Paste this into any AI assistant:

Play the other side of a hard conversation with me. The situation: Leading a major incident as the manager (not the fixer). Your job is to coordinate, and everyone wants you to fix. You are the person across the table and I am their manager. Invent a believable person and keep what is really on your mind to yourself until I ask well. After eight exchanges, tell me what I missed.

Rehearse in the app

Sources

  • Betsy Beyer, Chris Jones, Jennifer Petoff & Niall Richard Murphy (eds.), Site Reliability Engineering (2016), chapter 14, "Managing Incidents": incident commander, operations lead and communications lead roles. Free at sre.google.
  • Betsy Beyer et al. (eds.), The Site Reliability Workbook (2018), chapter 9, "Incident Response": worked examples of the roles, and agreeing the communication channel in advance.
  • PagerDuty, Incident Response Documentation (response.pagerduty.com): the incident commander role and training, including that the commander coordinates rather than fixes.

Published by Polanyi Works · Updated