Incident Readiness: Deciding Things Before You Need To

Incidents are a poor environment for decision-making. People are stressed, information is incomplete, and the cost of a wrong move is at its highest.

Preparation is simply deciding in advance what will happen, so that during an incident you are executing rather than inventing.

What must exist before an incident

Access that survives the failure. If your only administrative path depends on the system that is down, you have no path. A break-glass credential, stored securely offline, with its use alarmed, is not paranoia — it is the difference between responding and watching.

A current inventory. What runs, where, who owns it, what it depends on. During an incident is the wrong time to discover you do not know what a service talks to.

A communication channel independent of your infrastructure. If your chat platform runs on the affected system, you need somewhere else to coordinate.

Contact details that are reachable. Provider support, the on-call rota, and anyone who must be told. Printed or stored somewhere that does not require the affected system.

A tested restore. An untested backup is a hypothesis, and an incident is a poor time to test it.

The response sequence

1. Declare. Say out loud that this is an incident and name one incident lead. Ambiguity about who is coordinating wastes the first twenty minutes of every incident.

2. Assess. What is affected, who is affected, is it getting worse. Resist diagnosing before you have scoped.

3. Communicate early. A holding message acknowledging the problem is better than silence, even with no detail. Silence is read as absence.

4. Contain before you fix. Stop the bleeding. Roll back the deployment, disable the feature, fail over, block the source. Root cause analysis comes later.

5. Recover. Restore service, verify it properly, watch for recurrence.

6. Review without blame. What happened, why it was possible, what will change. Blameless review gets honest information; blameful review gets defensive information.

Suspected credential compromise

This deserves its own sequence because the instinct — change the password — is insufficient and can destroy evidence.

1. Preserve logs first. Export the audit trail before anything changes. On AWS that is CloudTrail, on Google Cloud the Audit Logs, on Azure the Activity Log. Once you start revoking, you lose the ability to see what happened.
2. Revoke, do not just rotate. Deactivate every access key, service account key and service principal secret. A rotated key with an old one still active achieves nothing.
3. Terminate active sessions. A password change does not always invalidate existing sessions.
4. Check for persistence. Attackers create new identities. Look for: new IAM users, new access keys on existing identities, new roles, modified trust policies allowing external accounts, new service accounts, altered recovery email and phone.
5. Check every region for resources you did not create. Cryptomining instances in unused regions are the classic finding.
6. Review billing for unexpected charges, which frequently reveals activity before the logs are read.
7. Re-enrol MFA and replace recovery options.
8. Then rebuild. If you cannot establish what was changed on a host, redeploy from a known-good image rather than cleaning it.

Step four is the one that decides whether the incident is over. Changing the password while an attacker-created access key remains active resolves nothing. The credential guide covers the underlying hygiene.

Data breach obligations

If personal data may have been accessed, legal obligations attach and they have short clocks. Under GDPR the supervisory authority must generally be notified within 72 hours of becoming aware, and affected individuals told without undue delay where the risk is high. Other jurisdictions have their own timelines.

Decide in advance who makes that assessment and who has authority to notify. That is not a decision to make at 2 a.m. for the first time. If you process personal data and have never had this conversation, have it before you need it.

Provider-side incidents

Not every incident is yours. When the provider is at fault:

• Check the provider's status page, and be aware it frequently lags reality.
• Check independent sources — outage trackers and community reports often lead the official page.
• Open a support ticket to register impact, even if the cause is known.
• Communicate to your users in your own words. "Our provider is having an issue" is still your outage from their perspective.
• Record the impact for the post-incident review, including whether your architecture could have tolerated it.

That final point often produces the most valuable follow-up action: a single-region, single-zone deployment will share every regional incident, and knowing that was a deliberate choice is different from discovering it during one.

Frequently asked questions

How big does a team need to be to have an on-call rota? Any team responsible for something that must stay up. Even a rota of two is better than an implicit assumption about who responds.

Should incident reviews be public? Internally always. Externally, a candid post-mortem builds more trust than a vague apology, and is worth considering for customer-affecting incidents.

What is the most common gap? No break-glass access path, and no tested restore. Both are cheap to fix and expensive to lack.

How do I practise? Run a tabletop exercise. Walk through a scenario verbally and find the gaps without breaking anything.

Provider support channels and status reporting differ; the comparison page covers what each platform in the catalogue offers.