create-runbook

Use AskUserQuestion:

• Header: "Task"
• Question: "What task do you want to automate? Describe it in a sentence or two — what goes in, what processing happens, and what comes out."
• Options:
  • "I'll describe it" / "Let me type a description" (user types in the text field)
  • "Show me examples first" / "Show example runbook tasks before I decide"

If "Show me examples first", display these real-world examples:

Example runbook tasks:

1. NL-to-SQL Regression — Pull failed queries from Langfuse, replay them against the NL-to-SQL API, execute on Snowflake, evaluate pass/fail, produce a regression report
2. PDF-to-Metadata Conversion — Extract metadata from academic PDFs, generate Croissant JSON-LD, validate against the schema, iterate on errors
3. Branded Social Graphics — Parse a text script, generate an AI image via Jetty workflow, compose HTML with text overlays, judge against a brand rubric, iterate
4. Clinical Training Content — Parse competency documents, generate training scenarios with rubric-scored quality, produce learning plans
5. Data Extraction Pipeline — Extract structured data from documents into multiple formats, validate schema compliance, produce quality report

Then re-ask the question (same AskUserQuestion, minus the "Show me examples" option).

Save the user's task description for use in all subsequent steps.

Default to rubric. Most runbooks produce content where quality is multi-dimensional, and rubric scoring is the more general-purpose pattern. Only choose programmatic when the task description clearly describes a coding/structured-output task. Skip the question entirely in those clear cases — just pick and tell the user what you picked and why (one line).

Pick programmatic without asking when the task description clearly involves any of:

• Schema validation, JSON Schema, OpenAPI, Croissant, JSON-LD
• SQL, query execution, database regression
• Unit tests, test suites, lint, type-check, build, compile
• API response shape checks, HTTP status assertions
• Code generation where the success criterion is "the code compiles / passes tests"
• File/data format conversion with a strict target spec (CSV with N columns, etc.)

Pick rubric without asking for everything else (content generation, creative output, reports, training material, image composition, summarization, classification quality, UX/brand checks).

Only fall back to AskUserQuestion when the task is genuinely ambiguous (e.g., "extract data from PDFs" — could be schema-validated or rubric-scored on completeness). When you ask:

• Header: "Evaluation"
• Question: "Your task could go either way. Programmatic = strict pass/fail against a schema or test. Rubric = 1–5 scoring across quality dimensions. Which fits?"
• Options:
  • "Quality rubric" / "Score against multiple criteria (1-5 scale)"
  • "Programmatic checks" / "Validate with code, schema, or tests (objective pass/fail)"

Save the chosen evaluation pattern: programmatic or rubric. When you skip the question, tell the user in one short line: "I'm using a rubric for this — your output is {reason}." or "Going with programmatic — {reason}."

Default to claude-code + anthropic/claude-sonnet-4.6 routed through OpenRouter (model_provider: openrouter) without asking. This is the right choice for the vast majority of users — strong reasoning, broad tool support, and a single OPENROUTER_API_KEY gives unified billing and provider failover. Only fall back to AskUserQuestion when the user has explicitly asked for a different agent in their task description (e.g., "route Claude through Anthropic directly", "use Codex", "I only have a Gemini key").

Before defaulting silently, do a quick check to confirm the user's collection has a usable key for the encouraged config. Run:

TOKEN="$(cat ~/.config/jetty/token)"
COLLECTION="{collection from Step 2a or jetty whoami default}"

# Org-level provider keys
COLL=$(curl -s -H "Authorization: Bearer $TOKEN" \
  "https://flows-api.jetty.io/api/v1/collections/$COLLECTION/environment")
read HAS_OPENROUTER HAS_ANTHROPIC <<< "$(echo "$COLL" | python3 -c "
import sys, json
evars = json.load(sys.stdin).get('environment_variables', {})
print('OPENROUTER_API_KEY' in evars, 'ANTHROPIC_API_KEY' in evars)
")"

# Trial status
TRIAL=$(curl -s -H "Authorization: Bearer $TOKEN" \
  "https://flows-api.jetty.io/api/v1/trial/$COLLECTION")
TRIAL_ACTIVE=$(echo "$TRIAL" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('active', False))")

Decision (every path keeps the claude-code agent):

• HAS_OPENROUTER == True → encouraged path: claude-code + anthropic/claude-sonnet-4.6 + model_provider: openrouter. One line: "Using Claude Code on OpenRouter (anthropic/claude-sonnet-4.6) — your org has OPENROUTER_API_KEY set."
• No OpenRouter key, but HAS_ANTHROPIC == True or TRIAL_ACTIVE == True → fall back to claude-code + claude-sonnet-4-6 + model_provider: anthropic. One line: "Using Claude Code via Anthropic (claude-sonnet-4-6) — your trial covers it." or "…your org has ANTHROPIC_API_KEY set."
• Neither → still default to the encouraged OpenRouter path, but tell the user: "Defaulting to Claude Code on OpenRouter (anthropic/claude-sonnet-4.6). Add an OPENROUTER_API_KEY in Jetty before running — or tell me to route Claude through Anthropic instead."

Only use AskUserQuestion when the user's task description explicitly names a different runtime, or they push back on the default. When you do ask:

• Header: "Agent Runtime"
• Question: "Which agent will run this runbook on Jetty?"
• Options:
  • "Claude Code on OpenRouter (Recommended)" / "Runs the Claude Code agent on anthropic/claude-sonnet-4.6 routed through OpenRouter. One OPENROUTER_API_KEY — unified billing and provider failover."
  • "Claude Code via Anthropic" / "Runs Claude Code on claude-sonnet-4-6 directly via Anthropic. Requires an ANTHROPIC_API_KEY."
  • "Codex (OpenAI)" / "Uses gpt-5.5 — strong at code generation. Requires an OPENAI_API_KEY."
  • "Gemini CLI (Google)" / "Uses gemini-3.1-pro-preview — free tier available. Requires a GOOGLE_API_KEY."

(The opencode agent on OpenRouter is also supported — see the agents-and-models reference — but Claude Code is the default runtime.)

Save the agent, model, and provider choice. The mapping is:

• Claude Code on OpenRouter → agent: claude-code, model: anthropic/claude-sonnet-4.6, model_provider: openrouter
• Claude Code via Anthropic → agent: claude-code, model: claude-sonnet-4-6, model_provider: anthropic
• Codex → agent: codex, model: gpt-5.5, model_provider: openai
• Gemini CLI → agent: gemini-cli, model: gemini-3.1-pro-preview, model_provider: google

Then pick the sandbox. Don't ask if the task obviously needs a browser — just pick prism-playwright and tell the user in one line. Browser is obvious when the task description mentions any of: web scraping, scrape, screenshot, browser, Playwright, Selenium, OAuth flow, web UI testing, e2e, HTML rendering, page navigation, crawl/crawler, login flow, or specific URLs/web apps.

If the task is clearly text/data-only (no browser cues), default to python312-uv and tell the user in one line.

Only ask when ambiguous (e.g., "fetch data from a website" — could be HTTP scraping or browser scraping). When you ask:

• Header: "Sandbox"
• Question: "Will your runbook need a real browser (Playwright + Chromium), or is HTTP enough?"
• Options:
  • "Browser" / "Use prism-playwright snapshot (Playwright + Chromium pre-installed)"
  • "No browser" / "Use python312-uv snapshot (lighter, faster startup)"

Save the snapshot choice. These values will be written into the runbook frontmatter in Step 3.

The runbook file will always be written as ./RUNBOOK.md in the current working directory — do not ask the user where to save it.

If you are running inside a Jetty authoring sandbox — detected by either AUTHORING_MISSION being set in the environment or /app/MISSION.md existing on disk — skip sub-steps 4d (Dependencies), 4h (Common Fixes), and 4i (Tips). These three sections are best filled in after the runbook has been executed at least once, when real failure modes, real dependencies, and real gotchas are known. Leave the placeholder rows in RUNBOOK.md as-is; they signal "fill in after first run". Walk through 4a, 4b, 4c, 4e, 4f, 4g only.

if [ -n "${AUTHORING_MISSION:-}" ] || [ -f /app/MISSION.md ]; then
  AUTHORING_SANDBOX=1
else
  AUTHORING_SANDBOX=0
fi

Outside the authoring sandbox (local CLI / IDE), walk through all nine sub-steps below.

Show the user the Objective section you drafted. Use AskUserQuestion:

• Header: "Objective"
• Question: "Here's the objective I drafted:\n\n{show the objective text}\n\nDoes this capture your task accurately?"
• Options:
  • "Looks good" / "Move on to the next section"
  • "Needs changes" / "Let me refine it" (user types corrections)

If they want changes, apply via Edit and move on.

Show the proposed output manifest. Use AskUserQuestion:

• Header: "Output Files"
• Question: "These are the files the runbook will produce:\n\n{list the files}\n\nDoes this look right?"
• Options:
  • "Looks good" / "This manifest is correct"
  • "Add a file" / "I need an additional output file"
  • "Change a file" / "One of these needs to be different"

Apply changes via Edit. Ensure validation_report.json and summary.md always remain in the manifest. When the user adds, removes, or renames a headline deliverable, update the primary_outputs: frontmatter to match — its first entry should be the file the user considers the main result (it becomes the "Main output" surfaced in spot).

Show proposed parameters. Use AskUserQuestion:

• Header: "Parameters"
• Question: "These are the configurable inputs:\n\n{list parameters}\n\nAnything to add or change?"
• Options:
  • "Looks good" / "These parameters are sufficient"
  • "Add more" / "I need additional parameters" (user describes them)

Apply changes via Edit.

Skip this sub-step entirely if AUTHORING_SANDBOX=1 (see Step 4 shortcut). Leave the Dependencies table placeholder row intact for later.

Use AskUserQuestion:

• Header: "Dependencies"
• Question: "What does your runbook need beyond the base environment?"
• Options:
  • "Jetty workflows" / "I'll call Jetty workflows as sub-steps"
  • "External APIs" / "I call non-Jetty APIs (REST, GraphQL, etc.)"
  • "Python/Node packages" / "I need specific libraries installed"
  • "None" / "No special dependencies — just standard tools"

For each selected category, ask a follow-up for specifics (workflow names, API URLs, package names). Populate the Dependencies table and the Step 1 setup script via Edit.

Use AskUserQuestion:

• Header: "Secrets"
• Question: "Does this runbook need any API keys, tokens, or other credentials?"
• Options:
  • "Yes" / "I need to declare secrets for API keys or credentials"
  • "No" / "No sensitive parameters needed"

If yes, for each secret collect via AskUserQuestion:

• Logical name (e.g., OPENAI_API_KEY)
• Collection environment variable name (usually same as logical name)
• Description
• Required or optional

Populate the secrets block in frontmatter with the collected values. For example:

secrets:
  OPENAI_API_KEY:
    env: OPENAI_API_KEY
    description: "OpenAI API key for embeddings"
    required: true

Also add a verification block in Step 1 (Environment Setup) that checks each required secret is available as an environment variable.

Multimodal pre-fill. If the task description involves images, video, vision, OCR, screenshots, image generation, or similar multimodal work, pre-populate the secrets: block without asking — and tell the user which keys you added in a single line. Pick from:

• REPLICATE_API_TOKEN — Flux, Seedance, Sora, segmentation, embeddings. The default choice for image/video generation.
• GEMINI_API_KEY — Imagen image generation, Gemini vision inputs.
• OPENAI_API_KEY — DALL·E, gpt-image-1, GPT-4 vision.

You only need to declare the ones the task actually uses (e.g., a "generate product hero images" task gets REPLICATE_API_TOKEN alone; "describe what's in this PDF" gets GEMINI_API_KEY or OPENAI_API_KEY). These three keys are also auto-forwarded by mise when present on the trajectory, so a trial-eligible collection will get usable tokens even without the explicit declaration — but declaring them keeps the runbook self-documenting and makes the verification block in Step 1 catch missing setup early.

Based on the task description, propose a sequence of processing steps. Show the user your proposed outline. Use AskUserQuestion:

• Header: "Processing Steps"
• Question: "Here's the step sequence I'm proposing:\n\n{numbered list of steps}\n\nWant to adjust?"
• Options:
  • "Looks good" / "This sequence works"
  • "Add a step" / "I need an additional step"
  • "Change order" / "The steps need reordering"
  • "Remove a step" / "One of these isn't needed"

Apply changes via Edit. For each confirmed step, write a skeleton with:

• Step name as header
• 2-3 sentence description of what to do
• Placeholder for API calls or code snippets: {TODO: add API call examples and expected response format}
• Placeholder for error handling: {TODO: add error handling for common failures}

This is the most important section. Branch based on the evaluation pattern:

For programmatic:

Use AskUserQuestion:

• Header: "Pass/Fail Criteria"
• Question: "Define what PASS, PARTIAL, and FAIL mean for your outputs. What makes an output correct? What makes it partially correct? What's a failure?"
• Options:
  • "I'll define them" / "Let me describe each status" (user types)
  • "Use defaults" / "Keep the template defaults and I'll refine later"

If they define criteria, update the status table via Edit.

For rubric:

Use AskUserQuestion:

• Header: "Rubric Criteria"
• Question: "What criteria matter for your output quality? Name 3-7 dimensions you'd score on a 1-5 scale.\n\nExamples: accuracy, completeness, clarity, brand compliance, technical correctness, creativity, formatting"
• Options:
  • "I'll list them" / "Let me name my criteria" (user types)
  • "Use 5 defaults" / "Start with generic criteria and I'll customize later"

If they provide criteria, build the rubric table with rows for each. For each criterion, ask (in a single AskUserQuestion):

• Header: "Rubric Details"
• Question: "For each criterion, briefly describe what 5 (excellent) and 1 (poor) look like. Or just list the criteria names and I'll draft reasonable descriptions.\n\n{list their criteria}"
• Options:
  • "I'll describe them" / "Let me define the scale for each" (user types)
  • "You draft them" / "Write reasonable descriptions and I'll review"

Update the rubric table via Edit.

Skip this sub-step entirely if AUTHORING_SANDBOX=1 (see Step 4 shortcut). Leave the Common Fixes table placeholder rows intact — they get filled in after the first real run surfaces actual failure modes.

Use AskUserQuestion:

• Header: "Common Fixes"
• Question: "Do you know the typical failure modes for this task? If so, describe them and I'll build a fix table. If not, you can fill this in after your first few runs."
• Options:
  • "I know some" / "Let me describe common issues" (user types)
  • "Skip for now" / "I'll fill this in after running the runbook"

If they provide issues, populate the Common Fixes table via Edit. If skipped, leave the placeholder rows.

Skip this sub-step entirely if AUTHORING_SANDBOX=1 (see Step 4 shortcut). Leave the Tips section's placeholder bullets intact — they get filled in after the first run reveals real gotchas.

Use AskUserQuestion:

• Header: "Tips"
• Question: "Any domain-specific gotchas, API quirks, or hard-won lessons you want to capture? These help the agent avoid known pitfalls."
• Options:
  • "Yes" / "I have some tips to add" (user types)
  • "Skip" / "Nothing comes to mind — I'll add tips later"

If they provide tips, update the Tips section via Edit.