Monitoring and alerting
Which signals deserve to wake you, which belong on a dashboard, how to set thresholds that survive a busy day, and how to keep an alert channel from going numb.
Open the procedureMonitoring: seeing a run go wrong before it finishes
Most automation failures are not dramatic. A run keeps executing against a stale configuration, or stops quietly while every dashboard still looks green, and nobody notices until the balances are reconciled. Monitoring exists to shorten the gap between something going wrong and someone knowing, and the useful measure of it is that gap, not the number of charts.
This section covers what to alert on and where to route it, what to record so a failure can still be explained weeks later, and the ordered first response when a key or an endpoint stops being trustworthy. The procedures that produce those events are in runbooks.
Test of an alert Someone has agreed, by name, to answer it
Test of a log It answers the question you have in three weeks
Test of a response The order is decided before the incident, not during
3 procedures in this section
Signals worth alerting on, log lines worth keeping, and the ordered response for the moment a key or an endpoint is no longer trustworthy.
Reading bot logs
A log schema that answers questions months later, the fields that make a failed transaction diagnosable, and what must never be written to a log file.
Open the procedureIncident response playbook
The ordered response to a suspected key compromise: contain, move what can still be moved, revoke, reconstruct the timeline, then rebuild on clean keys.
Open the procedureHow this section is used
Read alerting first, because it decides what the logs need to contain. Read the log schema second, because it decides whether an incident can be reconstructed at all. Read the incident playbook before you need it, once, calmly, and then keep it where you can open it on a phone.
Runbooks Numbered procedures with preconditions, rollback points and stop conditions. Funding a fleet, budgeting account rent, clearing a preflight and shutting a run down without stranding value. Key hygiene How keys are generated, where they live, what each one is allowed to touch, and how you rotate them without losing access to the accounts they control.