Reading Bot Logs: The Fields That Make a Failure Diagnosable

A bot log becomes diagnosable when every event is one line of JSON with the same field names every time. The fields that carry the diagnosis are the timestamp with a timezone, a run id, a wallet label, the venue, the action, the amounts, the transaction signature, the slot, the result and an error class. Prose lines cannot answer those questions later.

Everything else in this runbook follows from that choice. A stable schema is what makes filtering possible, what lets an alert carry a pointer into the log instead of a vague description, and what turns a pile of lines into a timeline you can hand to somebody else. Free-form logging feels faster on the first evening and costs you the answer on the night it matters.

One line per event, one stable schema

Newline-delimited JSON means each event is a complete object on its own line, with no wrapping array and no line breaks inside a record. That format survives truncation, appends cheaply, streams through standard tools without a parser, and stays readable if the process dies halfway through a run. When the file ends mid-line you lose one event rather than the ability to read the file at all.

Stability matters more than completeness. A field that appears under three different names across two releases is worse than a field that was never written, because filters silently miss records instead of returning nothing. Choose the names once, write them for every event even when the value is null, and version the schema explicitly if you must change it. Adding a field is safe; renaming one quietly is not.

Granularity should be one line per meaningful event, not one line per function call. An attempt to swap produces a line when it is requested, a line when it is submitted, and a line when it resolves. That is enough to reconstruct order and duration without producing a file nobody will read. Anything finer belongs at debug level and should expire quickly.

The fourteen fields worth writing

Each field below earns its place by answering a question you will actually ask. If you cannot state the question a field answers, it is decoration and it makes every line longer for no return. The required column marks the fields without which a line cannot be traced back to anything; the optional ones are worth having but do not break the timeline when absent.

Two design choices in that table repay attention. Amounts are integers in base units rather than decimal strings, because the ledger works in base units and any conversion you perform for readability is a conversion you can get wrong in a way that is invisible later. Error class is a short enumerated value rather than a message, because you will want to count occurrences of a failure type, and free-text messages never group cleanly.

Run ids and wallet labels as correlation keys

The run id is the field that makes a shared log file usable. Generate it once when a run starts, attach it to every line the run produces including the ones written by helper processes, and put it in every alert the run triggers. An alert that carries a run id turns the first minute of an investigation into a single filter rather than a hunt through timestamps that may not agree between machines.

Wallet labels do the same job across the fleet dimension. Writing a label such as exec-04 rather than a public key keeps lines short, keeps them readable after the key behind that label is rotated, and keeps key material off the page entirely. Store the label-to-address mapping in your account map and resolve it when an outside reader needs the address. Thresholds and channels for the alerts that reference these ids are covered under monitoring.

Correlation also works backwards. When a preflight check writes its own lines under the same run id, the log tells you what was verified before the first signature was produced, which frequently explains a failure faster than the failure line does. The checks worth recording are listed in the runbook on preflight before a live run.

Levels and the info-only rule

Levels exist so that one file can serve two readers: the person watching a run and the person debugging a defect. Debug carries payload detail and intermediate state. Info carries the run narrative. Warn carries degradation that did not stop the work. Error carries a unit of work that will not complete without intervention. Those four are enough, and adding more usually means the boundaries were never agreed.

Discipline about warn is what keeps alerting useful. A retry that later succeeds is a warn; a retry that exhausts the policy is an error. When every transient blip is logged as an error, the error count stops meaning anything, and the person on call learns to ignore the channel that carries it. That drift is gradual and hard to reverse once the habit forms.

What must never reach a log line

Logs travel. They get copied into tickets, pasted into chats, shipped to aggregators, captured by backups and screenshotted for a colleague. Assume every line you write will eventually be read by somebody you did not choose. That assumption makes the redaction list short and non-negotiable, and it explains why the safe design is to make secrets unavailable to the logging code rather than to filter them on the way out.

From alert to root cause

An alert tells you that something crossed a threshold. The log tells you what happened. Moving between the two should be mechanical, and it is mechanical only if the alert carries a run id and the log carries the fields above. Work the sequence below in order and resist the urge to jump to a hypothesis, because the most expensive mistake in an investigation is fixing the second problem you noticed while the first one keeps running.

1. Freeze the scope. Note the alert time, the run id and the wallet label it names before you touch anything. If the run is still going and the failure is repeating, stop it before you investigate; a run that keeps burning fees while you read is a run that changes the evidence underneath you.
2. Filter the log to that run. Pull every line with the run id into a separate view and read it in timestamp order. You are looking for the first line whose result is not success, not the loudest one. Failures cascade, and the last error in a file is usually a consequence rather than a cause.
3. Classify before you explain. Read the error class on the first failing line and group the subsequent failures by the same field. A single class repeated across every wallet points at configuration or an endpoint; several classes on one wallet points at that wallet, its balance or its accounts.
4. Separate slow from rejected. Compare latency on the failing lines against the successful ones earlier in the run. High latency with a submission failure suggests the endpoint or the network; low latency with a rejection suggests the transaction was refused, and the reason is on chain rather than in your process.
5. Verify on chain. Take the signature from the last submitted attempt and confirm it independently rather than trusting the log to be the whole story. A line saying submitted with no matching confirmation line is exactly the ambiguity that on-chain verification resolves.
6. Write the finding next to the run id. Record the cause, the fix and the run id together in your notes before you restart anything. The next occurrence will look familiar, and the note is what turns familiarity into a two-minute diagnosis instead of a repeat investigation.

Venue is the field that makes this sequence work across a multi-market run. When a session spreads across several markets, the same wallet can fail on one route and succeed on another within seconds, and a log line without a venue makes those two events look like one flapping problem; running a volume bot on Solana DEXs only stays diagnosable if the venue and the route are recorded on every line rather than inferred from configuration. Recording it costs a short string per event and saves the step where you guess which market a failure belongs to.

Turning a signature into evidence

The signature field is what makes a log line checkable by somebody who does not trust your process. Every other field is your software describing its own behaviour; the signature points at a record neither of you controls. Write it as soon as a transaction is submitted rather than only on success, because the submitted-but-never-confirmed case is precisely the one you will need to investigate and the one an optimistic logger loses.

Collected signatures also stop being only a debugging artefact. A run's worth of signatures is a dataset: you can fetch each transaction, read what actually settled and compare it against what the log said was requested, which is how a claim about slippage or fee drag stops being an impression and becomes a number. That method is set out in more depth in this guide to measuring execution quality from transaction data, and it only works if the signature was written at submission time for every attempt, including the ones that failed.

Keep the slot alongside the signature. Clocks on hosts drift and log timestamps come from the machine that wrote them, so slot is the ordering field that holds when you compare your record against anyone else's. A public explorer such as Solscan resolves a signature without any credential, which makes it a reasonable second opinion when your own endpoint is the thing under suspicion.

Retention and volume

Retention is a trade between the questions you can still answer and the exposure you are still carrying. Structured lines are small and worth keeping for months; verbose debug output is large, collects accidental detail and should expire in days. Set the policy per level rather than per file, and rotate daily so that a file name alone tells you which day you are opening.

Log hygiene checklist

Run this list once when the schema is first written and again whenever a new field is added, a new destination is configured or a new process starts writing to the same file. Most redaction accidents arrive with a change rather than at the beginning, because the original author knew what not to write and the change did not.

• Every line is a complete JSON object with no embedded newlines and no wrapping array.
• Field names are identical across every process that writes to the file, including helper scripts.
• Timestamps carry an explicit offset rather than an implied local timezone.
• The logging code has no access to key material, so a mistake cannot write what it cannot see.
• Endpoint URLs are replaced by a nickname before any request is logged, including in error paths.
• Exception handlers do not dump raw configuration objects, which is where credentials usually escape.
• Files rotate on a fixed schedule and the naming makes the date obvious without opening them.
• A copy exists somewhere the bot host cannot delete, so a compromised host cannot erase its own trail.
• Retention is set per level, with debug expiring far sooner than info and error.
• Anything shipped to a third party has been reviewed field by field, not sampled.

The item about the logging code having no access to secrets is the one that does the most work. Filters are written by people who are imagining the fields that exist today, and they fail the first time an unexpected object is passed into an error path. Structuring the process so the logger holds a label and never a key removes an entire class of accident instead of catching it late.

Pulling a single run apart

These commands read an existing file and fetch a transaction by signature. Nothing here writes to the chain, so they are safe to run while you are still deciding what happened, though the tail command should be pointed at the file rather than at a live process you might disturb. Replace the bracketed placeholders with your own run id, date, signature and endpoint.

tail -f logs/run-<date>.jsonl
grep '"run_id":"<run-id>"' logs/run-<date>.jsonl
grep '"result":"error"' logs/run-<date>.jsonl | tail -n 20
jq -r 'select(.run_id == "<run-id>") | [.ts, .wallet, .venue, .action, .result] | @tsv' logs/run-<date>.jsonl
jq -r 'select(.result == "error") | .error_class' logs/run-<date>.jsonl | sort | uniq -c
solana confirm -v <signature> --url <rpc-endpoint>
solana transaction-history <pubkey> --url <rpc-endpoint>

Notice how little tooling this needs. A grep for the run id gives the timeline, a grep for the error result gives the failures, and a count of error classes gives the shape of the incident in one line of output. That is the practical return on a stable schema: the investigation runs on tools that are already installed, on a file you already have, without a query language or a service in the middle. Which of those signals deserve an alert in the first place is covered in monitoring and alerting.

Questions the desk gets asked