Reading My Agent's Transcript Like a Code Review

There is a routine on my machine that runs at 06:00 PT every weekday. It clones a repository, picks a topic, drafts a 900-word essay, opens a TypeScript file, prepends a new object to an exported array, commits, pushes. By the time I wake up the post is live.

The trade-off of running an agent on a cron is that you give up sitting next to it. You do not see what tool it chose, what file it read, what it decided not to do. The transcript becomes the only window into a run that already finished. If I do not read it, I find out about silent failures the same way the audience does, which is by noticing what is missing.

I started reading the transcripts as a code review about six weeks ago. The first three I read, I caught one bug per transcript. The bugs were not in the code the agent was producing. They were in the decision tree the agent was running.

The three things I actually look for

Tool-call choices that should have been something else.

The agent has access to a Read tool, a Grep tool, and a WebSearch subagent. When it wants to find where a function lives, the right tool is Grep. The transcript will sometimes show three Read calls on files the agent guessed at, finding nothing, and then a Grep call that finds the symbol immediately. That is a four-call solution to a one-call problem. The post still ships. The cost is latency, tokens, and the next time the agent has the same choice in a tighter context window it will run out of room.

I do not fix this in the agent. I fix it in the routine prompt. The CLAUDE.md gets one more line, something like "when locating a symbol, Grep first; Read is for files you already know exist."

Decisions to skip a step.

The routine has a Step 7 that says run git diff constants.ts | head -40 and eyeball the first 40 lines. Two transcripts ago the agent skipped it. The commit pushed. The post rendered fine. The reason it shipped is that the build did not break, and the build is the only check downstream of the commit. The reason I caught the skip is that I was reading the transcript anyway.

A skipped step is not a bug today. It is a bug the day the curly-quote check would have caught an apostrophe that esbuild does not like, the build breaks, and the post does not render. I added a tighter line to the routine: Step 7 is not optional, and if you skipped it, restart from Step 6.

Confident output that is not supported by the transcript.

This is the one I find scariest. The agent writes a paragraph claiming a specific number from a specific log file. The log file was never read. The number is invented. The shape of the prose is right; the citation is fabricated; the routine ships.

The check is mechanical. Every claim in the draft that includes a number, a date, or a path, search the transcript for a Read or Bash call against the file the claim points at. If the transcript does not show the read, the claim is unverified. If the claim is unverified and the prose is confident, the claim is invented.

I caught this twice. The first time the agent claimed a P&L number from the OpenClaw DB without ever running the DB query. The second time it referenced a commit hash that did not exist in the repo. Both got cut from the post before publish. Both would have shipped if I had skipped the transcript read.

The shape of the review

I do not read the full transcript. I read three slices.

The first slice is the tool-call list, top to bottom. I am not reading the inputs. I am reading the sequence. The questions are: did the order make sense, were there obvious detours, did the agent run a tool more than once for the same purpose.

The second slice is the gap between the last Read call and the first Write call. Everything between those two is the draft phase. If the draft makes a specific claim, the claim should reference something the agent read in the calls immediately above it. If the draft is confident about a number and the reads above it do not contain that number, the number is suspect.

The third slice is the end of the transcript. The last fifty lines tell me whether the agent committed cleanly, whether the push succeeded, and whether the post-run Telegram notification fired or stayed silent on purpose. If the last fifty lines do not end with a clean commit log and a successful push, the run failed and I have a manual cleanup task.

Total time per transcript: about three minutes. The cost of skipping it is whatever the worst case looks like multiplied by the rate it happens. For my agent the rate is about one bad decision per three runs. The cost is not high because I am the only audience. If I were running this for a customer the rate would be unacceptable.

What does not work

I tried a self-review pass where the agent grades its own transcript at the end of the routine. It catches the easy ones, like a tool-call ordering mistake, and misses the hard ones, like a fabricated number. A self-review pass cannot catch the case where the agent is confidently wrong, because that is the case where the agent thinks it is right.

I tried a second agent reviewing the first agent's transcript. That worked better for the easy categories and still missed fabricated numbers. The reviewer does not have access to the underlying files the writer claimed to read; it has access to the transcript, which contains the fabrication, which it cannot distinguish from a real read without going to the source.

The thing that catches a fabricated number is a human reading the transcript, holding a mental model of what the agent should have read in order to make the claim it made, and noticing the gap. That is code review. It does not delegate well.

The closing read

A cron-scheduled agent is a long-running junior engineer who never asks for help and never raises a hand when it is unsure. The transcript is the only artifact left when the shift ends. If you do not review the transcript, you find out about the bugs the same way the public does, which is to say after the fact and through an audience.

Three minutes a day. Three slices. One bug per three runs at the current rate. The routine ships better than it would otherwise. The agent gets a slightly tighter routine prompt every week. The transcript log becomes the unit of code review, which is the only honest place to put the review when the code itself is decided after the fact, by something that does not have a code reviewer in the loop.