Monitoring and Alerting: What Deserves to Wake You Up

What monitoring is actually for

Alert on the small set of signals that mean value is moving without your instruction, that a run is dead when the schedule says it should be alive, or that failures are climbing away from their own baseline. Everything else belongs on a dashboard you open deliberately. An alert nobody has agreed to answer is not monitoring, it is decoration with a notification sound attached.

The purpose of a rule is to change what a person does in the next ten minutes. That test is stricter than it sounds, and most candidate rules fail it. A message that arrives, gets read, and produces no action was not an alert, it was a status line delivered at an inconvenient hour. Write the first response next to the rule before you enable it, and delete any rule where you cannot write one honestly.

Operators tend to build monitoring in the wrong order. The instinct is to collect metrics first and decide later what matters, which produces a wall of charts and no wake-up rule. Invert it. Decide the two or three outcomes that would make you stop the run, build the checks that detect exactly those, and add breadth afterwards. A narrow setup that fires correctly is worth more than a broad one that has been muted.

Three classes of signal

Separating signals into classes stops you from arguing about severity every time you add a rule. Liveness answers whether the thing is running. Correctness answers whether what it is doing matches what you configured. Value answers whether the balances are where you expect them to be. The three fail independently, and a setup that only covers one of them will miss the failures that hurt.

Most first attempts cover liveness only, because liveness is the easiest thing to measure. A process can be alive and healthy by every internal measure while submitting orders against the wrong mint, or while draining a wallet through fees on transactions that never land. Value checks catch what correctness checks miss, because they read the chain rather than the program, and the chain does not agree to be wrong on your behalf.

The alert severity matrix

Fix severity to the signal rather than to how you feel about it on the day. The matrix below assigns each signal a route and the shape of a threshold, not a number, because a number that is right for one venue and one order size is wrong for the next. Read the threshold column as a description of when a condition counts, and set the actual value from your own inventory and your own recent behaviour.

Notice that only four rows page a human. That ratio is deliberate. Paging is a scarce resource and every additional paging rule reduces the response you get from the rules that already exist. If you find yourself adding a fifth and a sixth paging rule, the honest move is usually to demote one of the earlier ones rather than to accept a louder night. Severity is a budget, and the budget does not grow because your setup did.

Standing up a minimal setup from nothing

The following order gets an operator from no monitoring to a defensible minimum without a platform, a vendor or a week of work. It builds the value check first because that is the check that would still matter if everything else were removed, and it ends with a test, because an untested route is an assumption wearing the costume of a control.

1. Write the inventory. List every wallet the run can sign with, every process that submits transactions, every endpoint in use and the schedule each process follows. Monitoring that cannot be traced back to an inventory line is monitoring you will not be able to interpret at three in the morning.
2. Define the end states you care about. Name the three or four outcomes that would make you stop: value leaving an account you did not instruct, a wallet under its floor, a dead process inside a scheduled window, failures climbing away from baseline. These become your paging rules, and nothing else does.
3. Build the balance check first. Schedule a job on a host separate from the bot that reads each wallet balance through your own endpoint and compares it against the floor in the inventory. Have it report every run rather than only on breach, so that a silent check is itself visible as a fault.
4. Add a liveness check. From that same external host, confirm the process is up during a scheduled window and that a transaction has landed within the interval the schedule implies. Treat absence of activity as a positive signal that something is wrong, not as an absence of news.
5. Add a failure-rate check. Use counters your process already emits, or derive them from the log, and compare the current window against the run recent baseline. Require a minimum number of events in the window before the rate is allowed to fire anything.
6. Create exactly two routes. One route makes a phone ring, for custody and value signals. One route lands in a channel a person reads within the working day. Hold the line at two until you have lived with them long enough to know what a third would carry.
7. Name one owner per alert. Each rule gets a person who has agreed out loud to answer it. Where no one will take that agreement, delete the rule rather than leaving it enabled and unowned, which is the state most dead monitoring ends up in.
8. Fire a test through every route. Trigger each route on purpose and confirm the message arrives on the device it is meant to reach. Put that test on a schedule, because routes break quietly through expired tokens, changed channels and phones that were replaced.
9. Write the first response next to the rule. One sentence per rule describing what the responder does first. The ordered response to a suspected key compromise is the destination for anything in the custody class, and the first response for those rules should simply point there.

Thresholds as shapes, not as numbers

Thresholds copied from someone else are the most common reason a setup produces noise. A failure rate that is normal for a busy venue at a small order size is alarming somewhere else, and a balance floor that suits a fan-out of ten wallets is meaningless for a fan-out of two hundred. Describe the shape of the condition, then fill in the value from your own inventory and your own history.

Three shapes cover almost everything. Any occurrence suits conditions where a single event is already unacceptable, such as an outbound transfer to an address that is not on the allowlist. Sustained over a window suits noisy signals where individual events are expected, such as endpoint errors. Relative to a recent baseline suits anything whose normal value moves with conditions, which includes almost every rate you will ever compute during a live run.

Windows also have to fit the schedule. A window that spans a period when the run is idle will always look healthy, because there are no events in it to be unhealthy. Anchor the window to activity rather than to wall clock time where you can, and where you cannot, suppress the rate rule outside scheduled windows and let the liveness rule carry that period instead. Two rules with clean boundaries beat one rule with an apology in its description.

Ownership, routing and the answer agreement

Routing is the part operators skip, and it is the part that decides whether any of this works. A rule needs a destination, a destination needs a person, and the person needs to have agreed. When those three are not in place, the rule still fires, the message still arrives, and the outcome is a shared assumption that somebody else is looking at it. Nobody is looking at it. That is the failure mode this section exists to prevent.

• Every rule has exactly one named owner, and the name is written next to the rule rather than held in memory.
• Each severity has one destination and only one, so that the paging route never carries anything a person can safely ignore.
• A test can be fired through every route on demand, without waiting for a real failure to prove the route works.
• Every rule carries a one-sentence first response, so the responder is never deciding from scratch at an unhelpful hour.
• Every rule has a review date, and a rule that reaches it without a defence is disabled rather than renewed by default.
• Alert bodies carry public keys, signatures, labels and counters, and never a seed phrase, a keypair file or an endpoint token.

Handover deserves the same discipline. When the person who agreed to answer is unavailable, the rule does not quietly become unowned. Either the ownership moves with the handover and the new owner agrees explicitly, or the rules in that class get disabled for the period and the run is paused. Anything else keeps the appearance of coverage while removing the thing that made it coverage.

Checks that do not depend on the bot

Independence is the property that separates real monitoring from a status page. A check is independent when it can report a failure of the thing it watches, which means it needs its own host, its own endpoint and its own route. Read balances from the chain rather than from the bot memory, confirm signatures through an endpoint the bot is not using, and run the schedule from a scheduler the bot cannot influence.

A hosted view of run state is genuinely useful for the first hour of an investigation, and something like the Solana volume bot pro console can put run state and recent failures in one place without any setup on your side, but a view rendered by the software doing the work cannot tell you that the software has stopped, so it does not remove the need for one alert that does not depend on that software being alive. Keep the independent check small enough that you never feel tempted to fold it into the main system for convenience.

solana balance <WALLET_PUBKEY> --url <RPC_URL>
solana address --keypair <KEYPAIR_PATH>
solana transaction-history <WALLET_PUBKEY> --url <RPC_URL> --limit 20
solana confirm -v <SIGNATURE> --url <RPC_URL>
pgrep -f <PROCESS_PATTERN>
systemctl is-active <SERVICE_NAME>
df -h <LOG_DIRECTORY>

Those are the primitives, not a finished monitor. Wrap them in whatever scheduler the external host already runs, compare the output against the values in your inventory, and send the result to a route you have tested. The command reference for the toolchain lives with the Agave client documentation, and it is worth checking flag names against it rather than against memory when you build the wrapper.

Alert fatigue and self-watching monitors

Two failure modes account for most dead monitoring. The first is fatigue, where a channel carries so much routine traffic that the one message that mattered arrives among things nobody reads. The second is self-reference, where the monitor lives inside the process it watches, so the failure it is least able to report is exactly the failure you built it for.

Fatigue is measurable without any special tooling. Keep a note of how often each rule fired and what the responder did, then read it at the review. Rules that produce action stay. Rules that produce a shrug get their threshold reshaped once, and if the next cycle looks the same they are demoted out of the paging route or removed. Nothing about this is sophisticated, but it is the only reliable defence against a channel that slowly stops being read.

What belongs on a dashboard instead

Plenty of useful information should never fire anything. Per-wallet balances over time, fee spend across a run, the mix of error codes, the distribution of confirmation times and the count of accounts you still hold all belong on a screen you open when you are already paying attention. They inform decisions on your schedule rather than interrupting you on theirs, and putting them behind a route trains you to ignore routes.

A useful division: if the correct response is something you would do this minute, it is an alert. If the correct response is something you would do this week, it is a dashboard row or a line in a digest. Fee spend usually sits in the second group, until it crosses a level where you would stop the run, at which point it becomes a rule with a floor rather than a chart with a trend. Move items between the groups deliberately, and write down when you moved them.

Dashboards also depend on the quality of what you write down in the first place. A chart that cannot be traced back to individual events is a chart you will not trust during an incident, which is when you need it most. The field schema in the log format that answers questions months later is the input side of everything on this page, and it is worth fixing before you build a second dashboard panel.

The review cycle that keeps rules honest

Rules rot. Endpoints change, wallets are retired, a schedule moves by an hour, a venue behaves differently, and a threshold that was reasonable at the time quietly becomes either noise or silence. Put a recurring review on the calendar, hold it even when nothing has broken, and give every rule an expiry so that renewal is an active decision rather than the default state of anything you once typed.

Keep the review short and structured. Walk the inventory and confirm each line still exists and is still watched. Walk the rules and, for each one, state when it last fired, what the responder did, and whether the owner is still the right person. Fire one test through each route while everyone is present. Then delete something. A review that never removes a rule is a review that is adding to the load every cycle without ever paying it down.

Tie the output back to the rest of the runbook. New failure classes discovered during a review usually belong in the pre-run checks rather than in a new alert, and the monitoring and incidents hub collects the pages that connect detection to response. The goal is not a bigger setup. The goal is a setup where every rule still has a person, a purpose and a first response you would be willing to follow at an hour you did not choose.

Questions the desk gets asked