Monitoring and Alerting: Signals Worth Waking Someone For

Most monitoring setups fail in the same way: they alert on everything, the team learns to ignore the alerts, and the one that mattered arrives in a stream of noise.

Good monitoring is less about collecting more and more about choosing what is worth interrupting someone for.

Alert on symptoms, not causes

This is the principle that fixes most alerting problems.

A high CPU alert is a cause. It may or may not mean anything is wrong — a batch job at 100% CPU is working correctly.

A "95th percentile response time exceeded two seconds for five minutes" alert is a symptom. Users are affected. That is worth an interruption.

Collect causes as metrics so you can diagnose. Alert on symptoms so you only interrupt when something is actually wrong.

The four signals

For any request-driven service, four signals cover most of what matters:

A sudden drop in traffic is one of the most reliable outage indicators there is, and almost nobody alerts on it. If requests fall to zero, either the world stopped caring or something between the user and your application broke.

Saturation: know your actual ceilings

Saturation matters only against a real limit. The ones that actually bite:

• Disk space. The most common preventable outage. Alert at 80%, page at 90%.
• Database connections. Frequently the real concurrency cap. Alert at 80% of max_connections.
• Memory with swap activity. Not memory usage — swap activity. The sizing guide explains why.
• CPU credit balance on burstable instances. Exhaustion produces sudden uniform slowness.
• Inode exhaustion. A disk with free space and no inodes behaves as if it is full.
• File descriptor limits on busy services.
• Cloud service quotas. Approaching an instance or IP limit stops deployments.
• Certificate expiry. Alert thirty days out. Expired certificates remain a leading cause of self-inflicted outages.

Thresholds that do not cry wolf

1. Require duration. "Above threshold for five minutes", not a single sample.
2. Use percentiles, not averages. An average hides the tail where the pain is.
3. Compare against a baseline where the metric is naturally variable.
4. Separate pages from tickets. Page for "users are affected now". Ticket for "this will matter next week".
5. Every alert needs a runbook. If the response is not known, the alert is not ready.
6. Delete alerts that have never been actionable. An alert nobody acts on is training the team to ignore alerts.

Logs, metrics and traces

• Metrics are cheap, aggregated and good for alerting. Numbers over time.
• Logs are expensive at volume and good for investigation. Retain what you will actually query, and set retention deliberately — verbose logging can rival compute in cost.
• Traces show a request's path across services. Worth the effort once you have more than a couple of services.

Start with metrics and structured logs. Add tracing when you can no longer tell which component is slow.

Structured logging

Log as structured data — JSON with consistent fields — rather than prose. The difference is between grep and query.

Every log line should carry: timestamp, level, service, request identifier, and where relevant a tenant or customer identifier. A request identifier propagated across services turns a distributed investigation from guesswork into a filter.

Never log credentials, tokens, full payment details or personal data beyond what you need. Logs are copied, exported and retained in places the original data never was, which makes them a data protection surface in their own right.

A minimum viable setup

For a small team, this is enough to be genuinely covered:

1. Uptime check from outside your infrastructure, every minute.
2. Latency and error rate on the application, alerting on symptoms.
3. Disk, memory and connection saturation alerts.
4. Certificate expiry alert at thirty days.
5. Cloud budget anomaly alert — covered here.
6. Structured application logs with a searchable backend and a deliberate retention window.
7. Alerts routed to where the team actually works.

That is a weekend of setup and it catches the overwhelming majority of incidents before users report them.

Frequently asked questions

How many alerts should a team have? Few enough that every one is taken seriously. If more than a handful fire weekly without action, prune.

Should monitoring run on the same infrastructure? At least one external check must not, or an outage takes your monitoring with it.

What about alert fatigue at night? Only page for things that genuinely cannot wait. Everything else creates a ticket. The distinction has to be enforced, not aspirational.

How long should logs be retained? Long enough to investigate a problem found late; short enough to control cost and data protection exposure. Thirty to ninety days is a common range, with longer retention only for what is required.

The signals above apply on any platform — browse the cloud account catalogue to see what each provider offers.