optimize-runbook

Confirm the three inputs are on disk. The trajectory metadata is at /app/trajectory.json (Mise also symlinks it under /app/trajectory/trajectory.json and the canonical /app/trajectory/{trajectory_id}.json — any of these works).

ls -la /app/RUNBOOK.md /app/baseline-RUNBOOK.md /app/trajectory.json

If any of those files is missing, emit the H3 JSON payload with patterns:["precheck failed: <which file>"] and proposed_changes:[], then stop. Never exit silently — Spot relies on the JSON to surface the failure to the user.

Read the runbook and the trajectory:

• /app/RUNBOOK.md — the runbook to edit (your working copy)
• /app/baseline-RUNBOOK.md — read-only snapshot for comparison
• /app/trajectory.json — the one trajectory you are analyzing
• /app/trajectory/ — step output assets (images, documents, uploaded inputs)

Do NOT call the Jetty trajectory list/fetch API. Do NOT use AskUserQuestion anywhere in headless mode.

From trajectory.json, extract and hold in memory:

• trajectory_id, status, total duration, iteration counts per step
• Each step's inputs, outputs, errors, duration
• Any labels already applied

Stay-alive cue. Mise's stream-tail heartbeat keys off log writes from your CLI. Long quiet stretches (e.g. >20s of pure thinking after a batch of Reads) used to starve the heartbeat and silently kill the activity mid-turn — your edits would still land on disk but Spot would lose the trail. The newer mise build heartbeats from a separate timer so this is no longer load-bearing, but prefer printing a one-line status before transitioning between phases (e.g. "Reads complete; analyzing patterns now.") regardless. It costs nothing, keeps the SSE stream fresh, and it gives the user a visible breadcrumb when the inter-tool gap is long.

Special case — degenerate baseline. If the trajectory's top-level status is anything other than completed, OR every step has status of null / failed / no outputs, the baseline never produced real evidence (e.g. the original agent failed at startup with Not logged in, an authentication error, or a quota issue). In that case, skip the six pattern lenses and go straight to H3 with patterns:["baseline trajectory produced no usable evidence: <one-line cause>"] and proposed_changes:[]. Do not fabricate patterns from log noise.

Otherwise, apply the same six pattern lenses as Step 4 of the interactive flow, against this single trajectory:

1. Consistent failures (evaluation criteria below threshold)
2. Iteration waste (retry-heavy steps)
3. Timeout / long-duration bottlenecks
4. Divergent agent behavior vs. the runbook's stated expectations
5. Missing guardrails (errors that the runbook's evaluation section doesn't catch)
6. Score plateaus (rubric-only — iterations that didn't improve)

For each pattern you find, plan one concrete edit to /app/RUNBOOK.md. Edits must be specific (exact before/after text) and cite the trajectory step(s) that motivated them.

Before editing any file, and before any other terminal action, print a single-line JSON object on its own line with this exact shape (Spot parses this for the diff-gutter citations):

{"type":"optimize_analysis","trajectory_id":"<id>","patterns":["<short description>","..."],"proposed_changes":[{"section":"<runbook section name>","before":"<exact current text>","after":"<exact replacement>","citations":["<step_name>","..."]},"..."]}

Rules:

• One JSON object, one line, no pretty-printing.
• citations must reference concrete step names or event ids from the trajectory. An empty citations array is a bug — don't emit one.
• Always emit this payload, even on the no-evidence path or precheck failure. When there's nothing to propose, emit "proposed_changes": [] and put the reason in "patterns" (one short string, e.g. "baseline trajectory failed at startup: Not logged in"). Stopping without emitting the payload leaves the user staring at a dead UI.

After the JSON line, print a short human-readable summary in prose (Spot will render it in the chat pane alongside the diff). Even when there's nothing to apply, the prose paragraph tells the user what you saw and what to do next (e.g. "Re-run the baseline with credentials configured, then start a new optimize session.").

For every item in proposed_changes, use the Edit tool against /app/RUNBOOK.md to replace the before text with the after text. Then bump the runbook version in the frontmatter (patch increment — e.g. 1.0.0 → 1.0.1). If no version field exists, add version: "1.0.1".

Do not edit /app/baseline-RUNBOOK.md — it stays pristine for the diff view.

Emit a one-paragraph summary (counts of patterns and edits applied). Then stop. The user may send follow-up turns asking for refinements (e.g. "also tighten step 3's iteration budget"). Apply those incrementally via Edit and wait again. Do not call the skill's subsequent steps; the interactive flow's Step 6–7 "apply changes" prompts are not applicable in headless mode.

Evaluation criteria scoring below threshold across multiple runs.

• Rubric: criteria scoring < 4 in more than half of runs
• Programmatic: stages showing FAIL/PARTIAL across runs

Steps that consistently need 2-3 retry rounds. Predictable first-attempt failures that could be prevented with better instructions or templates.

Runs that timed out or took disproportionately long. Which steps are the bottlenecks?

Cases where the agent interpreted instructions differently across runs. Structurally different outputs suggesting ambiguous instructions.

Errors not caught by evaluation criteria. Environment setup issues (wrong versions, missing tools).

Criteria that iterate but don't improve — suggesting the Common Fixes table lacks actionable guidance.

Present each pattern found with supporting evidence (trajectory IDs, scores, error messages).