All posts

Design SLO alerts with SLIs, error budgets, and burn rates

October 25, 2026 · 10 min read

A CPU threshold alone says little about user impact. Use an order API example to define eligible events and measurement points, calculate a 99.9% budget, combine burn-rate windows, and investigate through metrics and traces.

SLI, SLO, and SLA: what to define first

The order API is hypothetical; two seconds and 99.9% are not recommendations. Choose a target using user tolerance, order importance, dependency constraints, operating cost, and measured current performance. Avoid hiding failures of a high-value ordering path inside an average across unrelated operations.

Fix eligible events and measurement points first

Successes divided by all requests is underspecified. In this example, eligible events are legitimate order-creation requests. A correct order result returned within two seconds is good; other eligible outcomes are bad. Count timeouts and overload rejections caused by the service as bad. Decide how invalid input and user cancellations are handled in advance, rather than removing inconvenient events after an incident.

• Entry: define the operation, users, and measurement start. Fix whether each retry is a separate request or whether the unit is a business operation.
• Exit: define business success and the time bound beyond HTTP 2xx. For asynchronous acceptance, measure acceptance quality and the later completion deadline separately.
• Gaps: identify where to detect failures before the application and work that never finishes. Application completion counters alone miss entry-path failures.
• Aggregation: sum good, bad, and eligible counts across instances within the same window. Do not simply average per-Pod success percentages or p99 values.

Server-side measurement needs additional checks for browser connectivity failures and slow interactions. Synthetic checks can exercise a user path, but blindly mixing a few probes with a large real-user request population changes the metric's meaning. Define the role of each measurement and use them together.

Calculate a 99.9% error budget in events

An error budget is the amount of bad service compatible with the target. In an example with a fixed denominator of 100 million requests, a 0.1% allowed bad ratio permits 100,000 bad events. Observing 30,000 consumes 30%, leaving 70,000. This calculation assumes a known denominator; it does not make future request counts known partway through a month.

# Python 3: exact arithmetic for a hypothetical request-based SLO
from fractions import Fraction

slo = Fraction(999, 1000)
eligible_requests = 100_000_000  # fixed denominator for this example
allowed_bad = eligible_requests * (1 - slo)
observed_bad = 30_000
print(int(allowed_bad))                        # 100000
print(float(observed_bad / allowed_bad))       # 0.3 consumed
print(int(allowed_bad - observed_bad))         # 70000 remaining

error_ratio = Fraction(144, 10000)             # 1.44%
burn_rate = error_ratio / (1 - slo)
print(float(burn_rate))                       # 14.4
# Only with constant request volume across the 30-day period:
print(float(burn_rate * Fraction(1, 30 * 24))) # 0.02 of the budget

In a trailing 30-day window, old good and bad events leave as time advances, changing the denominator. Do not treat 70,000 as a permanently fixed balance. Request-based objectives also differ from time-based availability: a 0.1% bad-request allowance cannot directly become a downtime allowance.

A 1.44% bad-event ratio means a 14.4× burn rate

Burn rate is observed bad-event ratio / (1 - SLO). With a 99.9% objective, a 1.44% bad ratio gives 0.0144 / 0.001 = 14.4. Use the same definition across windows, but a 14.4× rate alone does not establish the fraction of the entire period's budget consumed.

Assuming constant eligible requests per hour throughout 30 days, or 720 hours, one hour at 14.4× consumes 14.4 / 720 = 2% of the period's budget. With varying traffic, check actual consumption as bad events in the window / ((1 - SLO) * eligible events in the full period). An overnight hour and a peak hour can consume different budget shares at the same bad ratio.

Combine long and short windows for notification

The Google SRE Workbook provides the following starting combinations for a 30-day, 99.9% objective. Adjust them to traffic and required response time rather than applying them universally. Time-based budget percentages use the traffic assumptions explained above.

# Pseudocode: ratios aggregate eligible request counts across instances
# Each window must have eligible traffic and valid, fresh telemetry.
fast = bad_ratio_1h > 0.0144 and bad_ratio_5m > 0.0144
slow = bad_ratio_6h > 0.0060 and bad_ratio_30m > 0.0060
page = fast or slow
# Missing telemetry and zero traffic need separate handling.

The long window checks accumulated impact; the short one checks whether degradation continues. Even a brief burst can alert if both windows cross their thresholds. A long window does not guarantee short incidents will be ignored. Test notification deduplication and recovery behavior too.

Separate sparse traffic, zero events, and missing data

One failure among ten hourly requests is a 10% bad ratio and a 100× burn rate for a 99.9% objective. A few failures cause large swings, so identical thresholds behave differently than in a busy service. For operations where one failure matters, add individual monitoring; use separate synthetic checks for ongoing path validation, matching the response to impact.

Narrow impact with metrics, then investigate traces

A notification should link to the affected operation, windows, good/bad counts, current and target values, recent changes, and an investigation procedure. Narrow impact using route-level latency, errors, traffic, and saturation, then inspect failures in logs and distributed traces. CPU and memory help diagnose causes; preventive checks for disk exhaustion and certificate expiry remain useful.

Each distinct metric-label combination adds a time series. In a hypothetical example, 20 routes × 5 outcomes × 10 Pods permit up to 1,000 combinations. Adding 10,000 users permits 10 million. Only observed combinations are created, but avoid unbounded user IDs and raw URLs; use route templates. Histogram buckets and related series need an additional estimate.

Keep trace IDs out of ordinary metric labels; use logs or supported exemplars to enter a trace. Propagate HTTP context through instrumentation supporting W3C Trace Context. Hops within a trace have different spans, so do not implement propagation by copying the entire traceparent value as an immutable identifier. Keep credentials and personal information out of diagnostic context.

Test what the person receiving an alert can do

1. Check firing and recovery against healthy traffic, sustained failure, a brief burst, sparse traffic, zero events, and missing telemetry.
2. Follow the alert to the affected path and investigation views, checking recent changes, external dependencies, and database waits.
3. Define available actions such as rollback, admission control, or dependency failover, plus ownership and escalation paths.
4. Agree how high budget consumption affects releases. Review missed incidents and unnecessary interruptions alongside notification counts.

• Google SRE — Service level indicators and objectives
• Google SRE Workbook — Alerting on SLOs
• Google SRE — Monitoring distributed systems
• Prometheus — Metric and label naming
• W3C — Trace Context