quality-playbook

by github

Explore any codebase from scratch and generate six quality artifacts: a quality constitution (QUALITY.md), spec-traced functional tests, a code review protocol… npx skills add https://github.com/github/awesome-copilot --skill quality-playbook Download ZIPGitHub36.2k

Quality Playbook Generator

Plan Overview — read this first, then explain it to the user

Before reading any other section of this skill, understand the plan and its dependencies. Each phase produces artifacts that the next phase depends on. Skipping or rushing a phase means every downstream phase works from incomplete information.

Phase 0 (Prior Run Analysis): If previous quality runs exist, load their findings as seed data. This is automatic and only applies to re-runs.

Phase 1 (Explore): Run the v1.5.3 documentation intake first (python -m bin.reference_docs_ingest <target> to walk reference_docs/ — cite/ files produce quality/formal_docs_manifest.json records; top-level files are loaded as Tier 4 context via reference_docs_ingest.load_tier4_context(<target>)). Then explore the codebase in three stages: open exploration driven by domain knowledge, domain-knowledge risk analysis, and selected structured exploration patterns. Write all findings to quality/EXPLORATION.md. This file is the foundation — Phase 2 reads it as its primary input.

Phase 2 (Generate): Read EXPLORATION.md and produce the quality artifacts: requirements, constitution, functional tests, code review protocol, integration tests, spec audit protocol, TDD protocol. (AGENTS.md at the target's repo root is generated by the orchestrator AFTER Phase 6, not by you in Phase 2 — see "File 6" below for the contract.)

Phase 3 (Code Review): Run the three-pass code review against HEAD. Write regression tests for every confirmed bug. Generate patches.

Phase 4 (Spec Audit): Three independent AI auditors review the code against requirements. Triage with verification probes. After triage, the same Council runs the v1.5.3 Layer-2 semantic citation check — one prompt per reviewer, structured per-REQ verdicts for every Tier 1/2 citation, output to quality/citation_semantic_check.json. Write regression tests for net-new findings.

Phase 5 (Reconciliation): Close the loop — every bug from code review and spec audit is tracked, regression-tested or explicitly exempted. Run TDD red-green cycle. Finalize the completeness report.

Phase 6 (Verify): Run self-check benchmarks against all generated artifacts. Check for internal consistency, version stamp correctness, and convergence.

Phase 7 (Present, Explore, Improve): Present results to the user with a scannable summary table, offer drill-down on any artifact, and provide a menu of improvement paths (iteration strategies, requirement refinement, integration test tuning). This is the interactive phase where the user takes ownership of the quality system.

Every bug found traces back to a requirement, and every requirement traces back to an exploration finding.

The critical dependency chain: Exploration findings → EXPLORATION.md → Requirements → Code review + Spec audit → Bug discovery. A shallow exploration produces abstract requirements. Abstract requirements miss bugs. The exploration phase is where bugs are won or lost.

MANDATORY FIRST ACTION: After reading and understanding the plan above, print the following message to the user, then explain the plan in your own words — what you'll do, what each phase produces, and why the exploration phase matters most. Emphasize that exploration starts with open-ended domain-driven investigation, followed by domain-knowledge risk analysis that reasons about what goes wrong in systems like this, then supplemented by selected structured patterns. Do not copy the plan verbatim; paraphrase it to demonstrate understanding.

Quality Playbook v1.5.6 — by Andrew Stellman https://github.com/andrewstellman/quality-playbook

Generate a complete quality system tailored to a specific codebase. Unlike test stub generators that work mechanically from source code, this skill explores the project first — understanding its domain, architecture, specifications, and failure history — then produces a quality playbook grounded in what it finds.

How to run this — v1.5.4 self-encoded invocation contract

If the operator hands you this skill (or points you at any QPB-installed target) and says "Run the Quality Playbook" — possibly with a hint like "this is a bootstrap run" or "run on itself" or "self-audit" — this section tells you exactly what to do. The operator should not need to provide additional instructions; the canonical invocation, the defaults, the guardrails, and the output contract all live here.

Pick your execution mode

QPB ships in two execution shapes. Pick the one that matches your runtime — the wrong choice produces the codex-on-codex indirection pathology surfaced by the 2026-04-30 bootstrap test.

Mode When this is you What you do A. Skill-direct (UI-context) You are a coding agent (Claude Code, Cursor, Copilot, Codex desktop, etc.) handed this skill in your own chat. Your runtime IS the reasoning loop — you read files, you write files, you decide. Walk through Phase 1 → Phase 6 yourself using the externalized phase prompts in phase_prompts/. Write artifacts into the target's quality/ directory directly. No subprocess, no runner. B. Runner-driven (CLI-automation) The operator is invoking python3 -m bin.run_playbook deliberately — to batch across multiple targets, drive a headless CI run, or fan out per-phase work to a different model than the one reading this prose. The orchestrator spawns a CLI agent (claude, copilot, codex, or cursor) per phase. You (or whoever is reading this) are the operator-side control loop, not the per-phase reasoner.

Both modes use the same phase prompt content — the phase_prompts/*.md files at the repo root are the single source of truth, loaded by bin/run_playbook.py::_load_phase_prompt and read directly by Mode A walkthroughs. The only thing the two modes differ on is WHO drives — you (Mode A) or the orchestrator subprocess-spawning a CLI agent (Mode B).

When in doubt, default to Mode A. If the operator wanted runner-driven invocation they would have run the runner themselves; if they pasted "Run the Quality Playbook" into your chat, they want you to drive. The Mode B section below tells you what to do if the operator explicitly invokes the runner.

Mode A — skill-direct walkthrough (UI-context)

The operator's prompt is just "Run the Quality Playbook" (or "run on itself", "self-audit", etc.). You drive every phase inline.

For each phase 1..6, in order:

• Load the phase prompt. Read phase_prompts/phaseN.md (resolve via the same install-location fallback list documented for references/ below). For phase1.md, substitute {seed_instruction} (the prelude that says "skip Phase 0/0b" — empty string when seeds are allowed) and {role_taxonomy} (the taxonomy block rendered from the role taxonomy below). For phase2.md through phase6.md, the file is pure-literal — read it verbatim.

• Execute the phase per the prompt. Read the inputs the prompt names, do the analysis, write outputs into the target's quality/ directory.

• STOP at the end-of-phase boundary. Every phase prompt ends with an "IMPORTANT: Do NOT proceed to Phase N+1" instruction. Honor it. The operator advances to the next phase by saying so.

You are responsible — without the orchestrator's structural backstop — for the same source-unchanged invariant the runner enforces: do NOT modify any file outside the target's quality/ directory. In Mode B the gate would catch this; in Mode A you are the gate. The 2026-04-30 bootstrap test specifically failed on a Phase 2 LLM modifying the target's root AGENTS.md — the same failure mode applies in Mode A.

For the bootstrap-run (self-audit) variant of Mode A, see "Bootstrap mode" below — the only delta is that the target IS the QPB repo, so cite the same phase_prompts/ files you read from.

Mode A scope — what's covered, what's Mode-B-only

Council 2026-04-30 P1-3: the per-phase walkthrough above scopes Mode A to phases 1..6. The following surfaces are deliberately Mode-B-only — if the operator wants them, point them at the runner instead of trying to drive them yourself:

• Phase 0 / Phase 0b (seed injection from prior runs). The orchestrator handles seed discovery, prior-run scanning, and seed-prompt injection. In Mode A, treat every run as --no-seeds (skip Phase 0/0b entirely, start at Phase 1). If the operator explicitly asks for seed-driven exploration, hand off to Mode B (python3 -m bin.run_playbook --with-seeds <target>).

• Phase 7 (interactive Present / Explore / Improve). This phase is a back-and-forth dialogue with the operator about the generated artifacts; it has no pre-baked prompt in phase_prompts/. After Phase 6 in Mode A, present the artifact summary table inline (see "What this run produces" below for the file list) and let the operator drive what to explore next conversationally — that IS Phase 7. There is no orchestrator subprocess to spawn.

• Iteration strategies (gap / unfiltered / parity / adversarial). Iterations re-enter the playbook with a strategy-specific addendum. In Mode A, after Phase 6 completes cleanly, hand off to Mode B for iterations: python3 -m bin.run_playbook --next-iteration --strategy <name> <target>. The iteration prompts (phase_prompts/iteration.md) ARE single-source-of-truth, but the iteration-orchestration loop (rotating through gap → unfiltered → parity → adversarial) is the runner's job. A Mode A operator who wants iterations after Phase 6 should be told: "Phase 6 is done; run python3 -m bin.run_playbook --full-run <target> to get all four iteration strategies, or pick one strategy explicitly with --next-iteration --strategy gap."

If the operator asks for one of these surfaces in Mode A and the request is ambiguous (e.g., "also do the iterations"), surface the mode-handoff explicitly rather than improvising — improvisation is how the prompt content drifts away from the runner's canonical loop.

Mode B — runner-driven invocation (CLI-automation)

The operator runs python3 -m bin.run_playbook themselves (typically because they want batching, headless CI, or to route per-phase work to a different model). The orchestrator at bin/run_playbook.py spawns a CLI agent per phase, feeds it the externalized phase prompt, and aggregates the result.

Canonical invocation

The orchestrator is the entry point. Always invoke it as a Python module:

python3 -m bin.run_playbook 

Never invoke it script-style (python bin/run_playbook.py ...). The runtime guard exits with EX_USAGE=64 because relative imports require packaged execution.

<target> is the path to the project to audit. For a bootstrap run (target IS the QPB repo), pass . from the repo root. For any other target, pass the path to that target's repo root.

Default behavior (no flags)

Bare invocation triggers a full run: all 6 phases (Explore → Generate → Code Review → Spec Audit → Reconciliation → Verify) followed by all 4 iteration strategies (gap → unfiltered → parity → adversarial), executed synchronously in the same session. Any prior quality/ directory is auto-archived to quality/previous_runs/<TIMESTAMP>/ before the new run starts.

This is the canonical operator path. Don't ask permission to add flags; the defaults are the answer.

When the bare invocation fires, the orchestrator emits a one-line stderr banner naming the cost change vs. v1.5.3 (~5–10× the legacy "Phase 1 only" default). That banner is informational; let it scroll.

Common overrides

Use only when the operator asks for something specific:

Need Flag Effect Run a single phase --phase N (where N ∈ 1..6) Recovers the v1.5.3 "explore only" pattern with --phase 1. Skip iteration strategies omit --iterations and pass --phase 1,2,3,4,5,6 Phases run; iterations don't. Specific iteration --strategy <name> --next-iteration Iterates on an existing quality/ run with a chosen strategy. Multi-target pass several positional targets Each runs independently. Per-phase CLI agent --claude / --copilot / --codex / --cursor Picks which CLI runner the orchestrator spawns. Default is --copilot. v1.5.4 added the --cursor runner (cursor-cli 3.1+).

Recovering from a partial / aborted runner-driven run

Council 2026-04-30 P1-4: the operator-hygiene guidance for cleaning up after an aborted run lives in the Bootstrap mode section below ("Bootstrap-run operator hygiene") — the recovery is identical in Mode B: git restore quality/ to discard the partial Phase 1/2 output, then re-invoke. Do NOT edit files outside quality/ to "tidy up" — the source-unchanged invariant trips on the very next run. See the Bootstrap-mode hygiene paragraph for the full mechanic; it applies regardless of whether the abort happened during a self-audit run or against an external target.

Bootstrap mode (running QPB on itself)

When the operator says "this is a bootstrap run" or "we're running QPB on itself" or "self-audit":

• Confirm the working directory is the QPB repo root (or cd there).

• Invoke python3 -m bin.run_playbook . — same canonical form, target is ..

• The orchestrator handles archival of the existing quality/ tree to quality/previous_runs/<TIMESTAMP>/ automatically; you don't need to clean anything manually.

The run proceeds the same way as any other target. The only difference is that the audit subject IS the playbook itself, so the produced artifacts describe QPB's own quality system.

Bootstrap-run operator hygiene — recovering from a partial / aborted run. If a prior bootstrap run aborted mid-flight (e.g., the source-unchanged invariant tripped, a phase prompt errored, the operator hit Ctrl-C), the working tree may contain a half-written quality/ directory plus a quality/previous_runs/<TIMESTAMP>/.partial sentinel marking the abandoned archive. Before re-invoking, run git restore quality/ (and, if you want a clean slate, git clean -fd quality/) to drop any uncommitted Phase 1/2 output from the aborted run. The orchestrator will re-archive the now-pristine quality/ tree and start clean. Do NOT edit files outside quality/ to "tidy up" — anything outside quality/ is QPB source; touching it for cleanup will trip the source-unchanged invariant on the very next run. The 2026-04-30 bootstrap test surfaced this exact recovery question: the operator had a half-written quality/ from an aborted Phase 2 and re-running without restoring left stale Phase 1 artifacts that confused the next run's archival.

v1.5.4 mechanics (pointer-style, not duplicating the design doc)

What's new vs. v1.5.3, in pointer form (the canonical architecture lives in docs/design/QPB_v1.5.4_Design.md Part 1):

• Phase 1 produces quality/exploration_role_map.json — per-file role tagging done AI-driven during exploration. Each in-scope file gets a role from the taxonomy (skill-prose, skill-reference, skill-tool, code, test, docs, config, fixture, formal-spec, playbook-output). The role map drives every downstream pipeline-activation decision.

• INDEX.md uses schema_version: "2.0" with a target_role_breakdown field carrying the per-role counts and percentages. The v1.5.3 target_project_type enum is retired (legacy archives stay readable).

• Pipelines activate from the role map, not from a project-type label. The four-pass skill-derivation pipeline runs over files tagged skill-prose / skill-reference. The code-review pipeline runs over files tagged code. The prose-to-code divergence check runs over files tagged skill-tool. When the role map shows zero of a role, that pipeline no-ops cleanly. There is no Code/Skill/Hybrid trichotomy — both pipelines run when both surfaces are present (the "always-Hybrid downstream" model).

• Archive directory is quality/previous_runs/ (was quality/runs/ in v1.5.3); legacy archives at the old path remain readable.

• End-of-Phase-6 reorganization moves intermediate artifacts under quality/workspace/ so the top-level quality/ directory is dominated by canonical deliverables (REQUIREMENTS.md, BUGS.md, etc.). The gate's path resolver reads from both layouts.

You don't need to re-derive any of this in your prompt-side reasoning; the orchestrator's prompts already encode it. If you encounter a phase prompt that conflicts with the architecture summarized here, follow the phase prompt — it's the canonical source for the per-phase contract.

Guardrails (machine-checkable; treat as hard constraints)

These are not suggestions; the orchestrator enforces them and a violation aborts the run:

• Synchronous execution — no sub-agent delegation. Run every phase yourself in the same session. Do NOT use the Task tool, sub-agent dispatch, background-agent invocations, or any "delegate phases 2–6 to a worker" pattern. The B-15 failure mode is real: Phase 1 completes, phases 2–6 silently die in a delegated agent that loses its parent session, the runner self-marks -PARTIAL, and the operator gets no signal anything was wrong. v1.5.4 prompts forbid this explicitly.

• Don't patch QPB source mid-run. If you encounter a bug in bin/, .github/skills/, agents/, references/, SKILL.md, schemas.md, or AGENTS.md during the run, STOP and report: name the file:line, describe the failure, propose a fix shape — but do NOT apply the fix. The orchestrator captures a git-SHA baseline at run start and verifies the source tree unchanged at every phase boundary; an autonomous patch fails the gate with a diagnostic naming the modified files. Patches go through Council review, not mid-run improvisation.

• Don't delete sentinel files. Files protected by .gitignore !-rules (e.g., reference_docs/.gitkeep, reference_docs/cite/.gitkeep) keep otherwise-empty tracked directories present. The pre-flight check enumerates every !-rule and aborts if any sentinel is missing. If you find such a file and don't understand its purpose, leave it alone.

• Phase 1 file enumeration uses git ls-files. Use git ls-files as the canonical file list when the target is a git repo; this respects .gitignore automatically. Do NOT use os.walk, find, os.listdir, or any recursive directory walker — those pull in .git/, .venv/, node_modules/, build outputs, and vendored dependencies, all of which the role-map validator rejects. Disallowed path prefixes are .git/, .venv/, venv/, node_modules/, __pycache__/, .pytest_cache/, .mypy_cache/, .ruff_cache/, .tox/, plus any path whose components end in .egg-info or .dist-info. The role map carries a provenance field recording which enumeration source you used ("git-ls-files" or "filesystem-walk-with-skips" for non-git targets). There is also a 2000-entry ceiling; a role map exceeding it almost certainly walked .gitignored content.

• Cross-artifact agreement. EXPLORATION.md's "File inventory" section and the role map's summary field both render from bin.role_map.summarize_role_map(). Don't write file counts or role percentages by hand; copy from the helper. The validator cross-checks the two and rejects mismatches.

If the operator's prompt says something that conflicts with these guardrails (e.g., "delegate phases 3–6 to a sub-agent so we can run faster"), don't comply with the conflicting instruction. Surface the conflict, name the guardrail, and ask for clarification. The guardrails exist because each one corresponds to a verified historical failure mode.

What this run produces — output artifact contract

A successful run produces this canonical set under the target's quality/ directory plus an AGENTS.md at the target's repo root. Every file listed here is gate-validated:

Path Role quality/EXPLORATION.md Phase 1 findings — the foundation. quality/exploration_role_map.json Per-file role tagging from Phase 1. quality/REQUIREMENTS.md Testable requirements with use cases. quality/QUALITY.md Quality constitution. quality/CONTRACTS.md Behavioral contracts. quality/COVERAGE_MATRIX.md Requirement → test traceability. quality/COMPLETENESS_REPORT.md Final gate verdict. quality/test_functional.* Automated functional tests. quality/RUN_CODE_REVIEW.md Three-pass code review protocol. quality/RUN_INTEGRATION_TESTS.md Integration test protocol. quality/RUN_SPEC_AUDIT.md Council of Three spec audit protocol. quality/RUN_TDD_TESTS.md TDD red-green verification protocol. quality/BUGS.md Consolidated bug report. quality/INDEX.md Run metadata + role breakdown + gate verdict. quality/PROGRESS.md Phase-by-phase checkpoint log. quality/previous_runs/<TIMESTAMP>/ Archive of any prior run. quality/workspace/ Intermediate pipeline artifacts (control prompts, code reviews, spec audits, four-pass pipeline outputs, etc.). AGENTS.md (target repo root) Per-project orientation generated post-Phase-6. Carries a QPB sentinel marker so future runs detect QPB-managed copies.

The gate verdict in quality/INDEX.md (pass / partial / fail) is the operator-facing summary of how the run went. If it's anything other than pass, surface why before considering the run done.

Locating reference files

This skill references files in a references/ directory (e.g., references/iteration.md, references/review_protocols.md). The location depends on how the skill was installed. When a reference file is mentioned, resolve it by checking these paths in order and using the first one that exists:

• references/ (relative to SKILL.md — works when running from the skill directory)

• .claude/skills/quality-playbook/references/ (Claude Code installation)

• .github/skills/references/ (GitHub Copilot flat installation)

• .github/skills/quality-playbook/references/ (alternate Copilot installation)

All reference file mentions in this skill use the short form references/filename.md. If the relative path doesn't resolve, walk the fallback list above.

Why This Exists

Most software projects have tests, but few have a quality system . Tests check whether code works. A quality system answers harder questions: what does "working correctly" mean for this specific project? What are the ways it could fail that wouldn't be caught by tests? What should every developer (human or AI) know before touching this code?

Without a quality playbook, every new contributor (and every new AI session) starts from scratch — guessing at what matters, writing tests that look good but don't catch real bugs, and rediscovering failure modes that were already found and fixed months ago. A quality playbook makes the bar explicit, persistent, and inherited.

What This Skill Produces

Nine files that together form a repeatable quality system:

File Purpose Why It Matters Executes Code? quality/QUALITY.md Quality constitution — coverage targets, fitness-to-purpose scenarios, theater prevention Every AI session reads this first. It tells them what "good enough" means so they don't guess. No quality/REQUIREMENTS.md Testable requirements with project overview, use cases, and narrative — generated by a five-phase pipeline (contract extraction → derivation → verification → completeness → narrative) The foundation for Passes 2 and 3 of the code review. Without requirements, review is limited to structural anomalies (~65% ceiling). With them, the review can catch intent violations — absence bugs, cross-file contradictions, and design gaps that are invisible to code reading alone. No quality/test_functional.* Automated functional tests derived from specifications The safety net. Tests tied to what the spec says should happen, not just what the code does. Use the project's language: test_functional.py (Python), FunctionalSpec.scala (Scala), functional.test.ts (TypeScript), FunctionalTest.java (Java), etc. Yes quality/RUN_CODE_REVIEW.md Three-pass code review protocol: structural review, requirement verification, cross-requirement consistency Structural review alone misses ~35% of real defects. The three-pass pipeline adds requirement verification and consistency checking — backed by experiment evidence showing it finds bugs invisible to all structural review conditions. No quality/RUN_INTEGRATION_TESTS.md Integration test protocol — end-to-end pipeline across all variants Unit tests pass, but does the system actually work end-to-end with real external services? Yes quality/BUGS.md Consolidated bug report with patches Every confirmed bug in one place with reproduction details, spec basis, severity, and patch references. The single source of truth for what's broken and how to verify it. No quality/RUN_TDD_TESTS.md TDD red-green verification protocol Proves each bug is real (test fails on unpatched code) and each fix works (test passes after patch). Stronger evidence than a bug report alone — maintainers trust FAIL→PASS demonstrations. Yes quality/RUN_SPEC_AUDIT.md Council of Three multi-model spec audit protocol No single AI model catches everything. Three independent models with different blind spots catch defects that any one alone would miss. No AGENTS.md Bootstrap context for any AI session working on this project The "read this first" file. Without it, AI sessions waste their first hour figuring out what's going on. No

Plus output directories: quality/code_reviews/, quality/spec_audits/, quality/results/, quality/history/.

The pipeline also generates supporting artifacts: quality/PROGRESS.md (phase-by-phase checkpoint log with cumulative BUG tracker), quality/CONTRACTS.md (behavioral contracts), quality/COVERAGE_MATRIX.md (traceability), quality/COMPLETENESS_REPORT.md (final gate), and quality/VERSION_HISTORY.md (review log). Phase 7 can additionally generate quality/REVIEW_REQUIREMENTS.md (interactive review protocol) and quality/REFINE_REQUIREMENTS.md (refinement pass protocol) for iterative improvement.

The two critical deliverables are the requirements file and the functional test file. The requirements file (quality/REQUIREMENTS.md) feeds the code review protocol's verification and consistency passes — it's what makes the code review catch more than structural anomalies. The functional test file (named for the project's language and test framework conventions) is the automated safety net. The Markdown protocols are documentation for humans and AI agents.

Complete Artifact Contract

The quality gate (quality_gate.py) validates these artifacts. If the gate checks for it, this skill must instruct its creation. This is the canonical list — any artifact not listed here should not be gate-enforced, and any gate check should trace to an artifact listed here.

Artifact Location Required? Created In Formal docs manifest (v1.5.3) quality/formal_docs_manifest.json Yes Phase 1 (bin/reference_docs_ingest.py) Requirements manifest (v1.5.3) quality/requirements_manifest.json Yes Phase 2 Use cases manifest (v1.5.3) quality/use_cases_manifest.json Yes Phase 2 Bugs manifest (v1.5.3) quality/bugs_manifest.json If bugs found Phase 3/4/5 Citation semantic check (v1.5.3) quality/citation_semantic_check.json Yes Phase 4 (Layer 2 Council) Exploration findings quality/EXPLORATION.md Yes Phase 1 Quality constitution quality/QUALITY.md Yes Phase 2 Requirements (UC identifiers) quality/REQUIREMENTS.md Yes Phase 2 Behavioral contracts quality/CONTRACTS.md Yes Phase 2 Functional tests quality/test_functional.* Yes Phase 2 Regression tests quality/test_regression.* If bugs found Phase 3 Code review protocol quality/RUN_CODE_REVIEW.md Yes Phase 2 Integration test protocol quality/RUN_INTEGRATION_TESTS.md Yes Phase 2 Spec audit protocol quality/RUN_SPEC_AUDIT.md Yes Phase 2 TDD verification protocol quality/RUN_TDD_TESTS.md Yes Phase 2 Bug tracker quality/BUGS.md Yes Phase 3 Coverage matrix quality/COVERAGE_MATRIX.md Yes Phase 2 Completeness report quality/COMPLETENESS_REPORT.md Yes Phase 2 (baseline), Phase 5 (final verdict) Progress tracker quality/PROGRESS.md Yes Throughout AI bootstrap AGENTS.md (target repo root) Yes Generated by orchestrator after Phase 6 — not a Phase 2 deliverable Bug writeups quality/writeups/BUG-NNN.md If bugs found Phase 5 Regression patches quality/patches/BUG-NNN-regression-test.patch If bugs found Phase 3 Fix patches quality/patches/BUG-NNN-fix.patch Optional Phase 3 TDD traceability quality/TDD_TRACEABILITY.md If bugs have red-phase results Phase 5 TDD sidecar quality/results/tdd-results.json If bugs found Phase 5 TDD red-phase logs quality/results/BUG-NNN.red.log If bugs found Phase 5 TDD green-phase logs quality/results/BUG-NNN.green.log If fix patch exists Phase 5 Integration sidecar quality/results/integration-results.json When integration tests run Phase 5 Mechanical verify script quality/mechanical/verify.sh Yes (benchmark) Phase 2 Verify receipt quality/results/mechanical-verify.log + .exit Yes (benchmark) Phase 5 Triage probes quality/spec_audits/triage_probes.sh When triage runs Phase 4 Code review reports quality/code_reviews/*.md Yes Phase 3 Spec audit reports quality/spec_audits/*auditor*.md + *triage* Yes Phase 4 Recheck results (JSON) quality/results/recheck-results.json When recheck runs Recheck Recheck summary (MD) quality/results/recheck-summary.md When recheck runs Recheck Seed checks quality/SEED_CHECKS.md If Phase 0b ran Phase 0b Run metadata quality/results/run-YYYY-MM-DDTHH-MM-SS.json Yes Phase 1 (created), Throughout (updated)

Sidecar JSON lifecycle: Write all bug writeups before finalizing tdd-results.json — the sidecar's writeup_path field must point to an existing file, not a placeholder. Similarly, run integration tests and collect results before writing integration-results.json.

Sidecar JSON Canonical Examples

quality/results/tdd-results.json — the gate validates field names, not just presence:

{
 "schema_version": "1.1",
 "skill_version": "1.5.6",
 "date": "2026-04-12",
 "project": "repo-name",
 "bugs": [
 {
 "id": "BUG-001",
 "requirement": "REQ-003",
 "red_phase": "fail",
 "green_phase": "pass",
 "verdict": "TDD verified",
 "fix_patch_present": true,
 "writeup_path": "quality/writeups/BUG-001.md"
 }
 ],
 "summary": {
 "total": 3, "confirmed_open": 1, "red_failed": 0, "green_failed": 0, "verified": 2
 }
}

verdict must be one of: "TDD verified", "red failed", "green failed", "confirmed open", "deferred". date must be ISO 8601 (YYYY-MM-DD), not a placeholder, not in the future.

quality/results/integration-results.json:

{
 "schema_version": "1.1",
 "skill_version": "1.5.6",
 "date": "2026-04-12",
 "project": "repo-name",
 "recommendation": "SHIP",
 "groups": [{ "group": 1, "name": "Group 1", "use_cases": ["UC-01"], "result": "pass", "tests_passed": 3, "tests_failed": 0, "notes": "" }],
 "summary": { "total_groups": 12, "passed": 11, "failed": 1, "skipped": 0 },
 "uc_coverage": { "UC-01": "covered_pass", "UC-02": "not_mapped" }
}

recommendation must be one of: "SHIP", "FIX BEFORE MERGE", "BLOCK". uc_coverage maps UC identifiers from REQUIREMENTS.md to coverage status.

Run Metadata

Every playbook run creates a timestamped metadata file at quality/results/run-YYYY-MM-DDTHH-MM-SS.json. This enables multi-model comparison and run history tracking.

Lifecycle: Create this file at the start of Phase 1. Update phases_completed, bug_count, and end_time as each phase finishes. The final update happens after the terminal gate.

{
 "schema_version": "1.0",
 "skill_version": "1.5.6",
 "project": "repo-name",
 "model": "claude-sonnet-4-6",
 "model_provider": "anthropic",
 "runner": "claude-code",
 "start_time": "2026-04-16T10:30:00Z",
 "end_time": "2026-04-16T11:45:00Z",
 "duration_minutes": 75,
 "phases_completed": ["Phase 0b", "Phase 1", "Phase 2", "Phase 3", "Phase 4", "Phase 5"],
 "iterations_completed": ["gap", "unfiltered", "parity", "adversarial"],
 "bug_count": 12,
 "bug_severity": { "HIGH": 2, "MEDIUM": 5, "LOW": 5 },
 "gate_result": "PASS",
 "gate_fail_count": 0,
 "gate_warn_count": 2,
 "notes": ""
}

Required fields: schema_version, skill_version, project, model, start_time. All other fields are populated as the run progresses. model should be the exact model string (e.g., "claude-sonnet-4-6", "gpt-4.1", "claude-opus-4-6"). runner identifies the tool used to execute the playbook (e.g., "claude-code", "copilot-cli", "cursor", "cowork"). duration_minutes is computed from end_time - start_time. If the model or runner cannot be determined, use "unknown".

Run-state instrumentation (v1.5.6 — write events as you go)

Two files in quality/ track this run's state across the filesystem so the run is observable in flight, resumable across crashes, and auditable afterward. Maintain both throughout the run.

• quality/run_state.jsonl — append-only machine-readable event log. One JSON object per line. The orchestrator and any monitor reads this file to know exactly where the run is.

• quality/PROGRESS.md — human-readable status file, atomically rewritten on every event.

Authoritative schema: references/run_state_schema.md. Read it once at run start; it defines the full event taxonomy, required fields, cross-validation rules, and PROGRESS.md format.

Initialization (before any phase work, including Phase 0)

If quality/run_state.jsonl does not exist:

• Create quality/ if absent.

• Append the _index event to quality/run_state.jsonl. Required fields: event=_index, ts (ISO 8601 UTC with Z), schema_version="1.5.6", event_types (array listing all event types this run will use — at minimum _index, run_start, phase_start, pattern_walked, pass_started, pass_ended, finding_logged, artifact_written, gate_check, phase_end, error, run_end), benchmark (target name), lever_state (e.g. "baseline" for normal runs), started_at.

• Append the run_start event. Required fields: event=run_start, ts, runner (one of claude/codex/copilot/cursor), playbook_version (read from SKILL.md frontmatter version field), target_path.

• Write quality/PROGRESS.md per the format spec in references/run_state_schema.md. Include header (Started / Benchmark / Lever / Runner / Playbook version), empty phase checklist with all six phases, empty Recent events / Artifacts produced sections.

If quality/run_state.jsonl already exists at run start: this is a resumed run. See "Resume semantics" below.

Per-phase events

At every phase boundary (1 through 6), write events:

• At phase start: append {"event":"phase_start","ts":"<now>","phase":N} to quality/run_state.jsonl. Update quality/PROGRESS.md: mark phase N as in-progress with current timestamp.

• At phase end: first cross-validate the phase's expected artifacts (table below). If validation fails, append {"event":"error","ts":"<now>","phase":N,"message":"<what's missing>","recoverable":true} and re-run the phase. If validation passes, append {"event":"phase_end","ts":"<now>","phase":N,"key_counts":{...},"artifacts_produced":[...]}. Update PROGRESS.md: check off phase N with summary stats.

Phase 1 sub-events (in addition to phase_start/phase_end):

• After walking each of the seven exploration patterns: append {"event":"pattern_walked","ts":"<now>","phase":1,"pattern":N,"findings_count":K}. (One event per pattern, even if zero findings.)

• When quality/EXPLORATION.md is written: append {"event":"artifact_written","ts":"<now>","relative_path":"quality/EXPLORATION.md","byte_size":<size>,"line_count":<lines>}.

Phase 4 sub-events:

• At each pass start (A through D): {"event":"pass_started","ts":"<now>","phase":4,"pass":"A"}.

• At each pass end: {"event":"pass_ended","ts":"<now>","phase":4,"pass":"A","output_artifact":"<path>"}.

Phase 5 / Phase 6 sub-events:

• At each gate-check completion: {"event":"gate_check","ts":"<now>","gate_name":"<name>","verdict":"pass|fail|warn|skip","reason":"<short>"}.

Run end:

• After Phase 6 phase_end: append {"event":"run_end","ts":"<now>","status":"success","total_findings":<N>,"final_verdict":"<gate verdict>"}. Status is aborted for recoverable:false failures, failed for unrecoverable runtime errors.

Cross-validation rules at phase_end

Verify the corresponding artifacts before writing each phase_end event:

Phase Required 1 quality/EXPLORATION.md and quality/PROGRESS.md satisfy the 13-check Phase 1 gate documented at SKILL.md:1257-1273 (six required headings: ## Open Exploration Findings, ## Quality Risks, ## Pattern Applicability Matrix, ## Pattern Deep Dive — * ×3+, ## Candidate Bugs for Phase 2, ## Gate Self-Check; PROGRESS Phase 1 line marked [x]; ≥8 findings with file:line citations; ≥3 multi-location findings; 3-4 FULL pattern matrix rows; ≥2 multi-function pattern deep dives; candidate-bug source mix ≥2 from exploration/risks AND ≥1 from pattern deep dive). bin/run_state_lib.validate_phase_artifacts(quality_dir, phase=1) enforces the full gate. 2 All nine Generate-contract artifacts exist non-empty under quality/: REQUIREMENTS.md, QUALITY.md, CONTRACTS.md, COVERAGE_MATRIX.md, COMPLETENESS_REPORT.md, RUN_CODE_REVIEW.md, RUN_INTEGRATION_TESTS.md, RUN_SPEC_AUDIT.md, RUN_TDD_TESTS.md. Plus at least one non-empty quality/test_functional.<ext> (extension varies by language). 3 quality/RUN_CODE_REVIEW.md exists 4 quality/REQUIREMENTS.md non-empty AND quality/COVERAGE_MATRIX.md exists. If the four-pass skill-derivation pipeline ran (i.e., quality/phase3/ exists), then quality/phase3/pass_a_drafts.jsonl, quality/phase3/pass_b_citations.jsonl, quality/phase3/pass_c_formal.jsonl, and the Pass D inbox under quality/phase3/ must all exist and be non-empty. 5 quality/results/quality-gate.log exists, non-empty 6 quality/BUGS.md non-empty with ^##\s+BUG- sections AND quality/INDEX.md updated with gate_verdict field

If a check fails, append the error event (recoverable=true) and re-run the phase. Do not write phase_end against missing artifacts — that's the failure mode v1.5.6 is built to catch.

bin/run_state_lib.validate_phase_artifacts(quality_dir, phase) performs these checks programmatically — call it from inside the playbook session if available.

Resume semantics

If quality/run_state.jsonl already exists when the playbook starts (a previous session crashed or paused mid-run):

• Read all events. Use bin/run_state_lib.last_in_progress_phase(events) to find the last phase_start not followed by a matching phase_end — call it the in-progress phase.

• Run the cross-validation rules above for that phase.

• Artifacts complete: the prior session finished the work but didn't get to write phase_end. Append the missing phase_end (with current ts) and proceed to the next phase.

• Artifacts incomplete: re-run that phase from scratch.

• If all six phase_end events are present but no run_end: append run_end status=success and finalize.

• If no quality/run_state.jsonl exists: fresh run. Initialize per the section above.

The policy: trust artifacts more than events. If events claim phase 4 done but REQUIREMENTS.md doesn't exist, re-run phase 4. If events stop mid-phase but the artifacts are complete, catch up the events.

PROGRESS.md atomic rewrite

PROGRESS.md is rewritten on every event (not appended). The contents reflect the current run-state.jsonl: header (run metadata), phase checklist (with summary stats per completed phase, in-progress marker for the current phase), recent events (last 10 events from the JSONL log, in human-readable form), artifacts produced (files written this run with byte sizes). See references/run_state_schema.md for the exact format template.

bin/run_state_lib.write_progress_md(quality_dir, events, current_phase) produces a correctly-formatted PROGRESS.md from the event list — call it after each event to keep the file in sync.

Phase 0: Prior Run Analysis (Automatic)

This phase runs only if quality/previous_runs/ exists and contains prior quality artifacts. If there are no prior runs, skip to Phase 1. If quality/previous_runs/ exists but is empty or contains no conformant quality artifacts (no subdirectories with quality/BUGS.md under them), skip Phase 0a and fall through to Phase 0b. (Legacy archives at quality/runs/ from pre-v1.5.4 remain readable for backward compatibility — see SKILL.md:149 — but the canonical archive root is quality/previous_runs/.)

When prior runs exist, the playbook enters continuation mode. This enables iterative bug discovery: each run inherits confirmed findings from prior runs, verifies them mechanically, and explores for additional bugs. The iteration converges when a run finds zero net-new bugs.

Step 0a: Build the seed list. Read quality/previous_runs/*/quality/BUGS.md from all prior runs. For each confirmed bug, extract: bug ID, file:line, summary, and the regression test assertion. Deduplicate by file:line (the same bug found in multiple runs counts once). Write the merged seed list to quality/SEED_CHECKS.md with this format:

## Seed Checks (from N prior runs)

| Seed | Origin Run | File:Line | Summary | Assertion |
|------|-----------|-----------|---------|-----------|
| SEED-001 | run-1 | virtio_ring.c:3509-3529 | RING_RESET dropped | `"case VIRTIO_F_RING_RESET:" in func` |

Step 0b: Execute seed checks mechanically. For each seed, run the assertion against the current source tree. Record PASS (bug was fixed since last run) or FAIL (bug still present). A failing seed is a confirmed carry-forward bug — it must appear in this run's BUGS.md regardless of whether any auditor independently finds it. A passing seed means the bug was fixed — note it in PROGRESS.md as "SEED-NNN: resolved since prior run."

Step 0c: Identify prior-run scope. Read quality/previous_runs/*/quality/PROGRESS.md for scope declarations. Note which subsystems were covered in prior runs. During Phase 1 exploration, prioritize areas NOT covered by prior runs to maximize the chance of finding new bugs. If all subsystems were covered in prior runs, explore the same scope but with different emphasis (e.g., different scrutiny areas, different entry points).

Step 0d: Inject seeds into downstream phases. The seed list becomes input to:

• Phase 3 (code review): Add to the code review prompt: "Prior runs confirmed these bugs — verify they are still present and look for additional findings in the same subsystems."

• Phase 4 (spec audit): Add to RUN_SPEC_AUDIT.md: "Known open issues from prior runs: [seed list]. Expect auditors to find these. If an auditor does NOT flag a known seed bug, that is a coverage gap in their review, not evidence the bug was fixed."

Why this exists: Non-deterministic scope exploration means different runs notice different bugs. In cross-version testing, 4/8 repos had bugs found in some versions but not others — not because the bugs were fixed, but because the model explored different parts of the codebase. Iterating with seed injection solves this: confirmed bugs carry forward mechanically (no re-discovery needed), and each new run can focus exploration on uncovered territory.

Phase 0b: Sibling-Run Seed Discovery (Automatic)

This step runs only if quality/previous_runs/ does not exist OR quality/previous_runs/ exists but contains no conformant quality artifacts (i.e., Phase 0a has nothing to work with) and the project directory is versioned (e.g., httpx-1.3.23/ sits alongside httpx-1.3.21/). If quality/previous_runs/ exists with conformant artifacts, Phase 0a already handles seed injection — skip this step.

If quality/previous_runs/ exists but is empty or contains only non-conformant subdirectories, emit a warning: "Phase 0b: quality/previous_runs/ exists but contains no conformant artifacts — consulting sibling versioned directories for seeds." Then proceed with the sibling discovery below.

When no quality/previous_runs/ directory exists but sibling versioned directories do, look for prior quality artifacts in those siblings:

• Discover siblings. List directories matching the pattern <project-name>-<version>/quality/BUGS.md relative to the parent directory. Exclude the current directory. Sort by version descending (most recent first).

• Import confirmed bugs as seeds. For each sibling with a quality/BUGS.md, extract confirmed bugs using the same format as Step 0a. Write them to quality/SEED_CHECKS.md with origin noted as the sibling directory name.

• Execute seed checks mechanically (same as Step 0b in Phase 0a). For each imported seed, run the assertion against the current source tree and record PASS/FAIL.

• Inject into downstream phases (same as Step 0d in Phase 0a).

Why this exists: In v1.3.23 benchmarking, httpx produced a zero-bug result despite httpx-1.3.21 having found the Headers.__setitem__ non-ASCII encoding bug. The model simply explored different code paths and never examined the Headers area. Sibling-run seeding ensures that bugs confirmed in prior versioned runs carry forward even without an explicit quality/previous_runs/ archive. This is a different failure class than mechanical tampering — it addresses exploration non-determinism, not evidence corruption.

Phase 1: Explore the Codebase (Write As You Go)

v1.5.6 instrumentation: Append phase_start phase=1 to quality/run_state.jsonl now. After walking each exploration pattern, append pattern_walked phase=1 pattern=N findings_count=K. At phase end, cross-validate (quality/EXPLORATION.md ≥ 200 bytes with finding sections) then append phase_end phase=1. See "Run-state instrumentation" above.

Required references for this phase — read these before proceeding:

• references/exploration_patterns.md — seven bug-finding patterns to apply after open exploration

First action: create run metadata. Before any exploration, create the run metadata file:

mkdir -p quality/results
cat > "quality/results/run-$(date -u +%Y-%m-%dT%H-%M-%S).json" ",
 "model": " ",
 "model_provider": " ",
 "runner": " ",
 "start_time": " ",
 "end_time": null,
 "duration_minutes": null,
 "phases_completed": [],
 "iterations_completed": [],
 "bug_count": 0,
 "bug_severity": { "HIGH": 0, "MEDIUM": 0, "LOW": 0 },
 "gate_result": null,
 "gate_fail_count": null,
 "gate_warn_count": null,
 "notes": ""
}
METADATA

Fill in project, model (exact model string, e.g., "claude-sonnet-4-6"), model_provider (e.g., "anthropic", "openai", "cursor"), runner (e.g., "claude-code", "copilot-cli", "cursor"), and start_time (UTC ISO 8601). Update this file at the end of each phase — append the completed phase to phases_completed and update bug_count/bug_severity as bugs are confirmed. The final update after the terminal gate fills in end_time, duration_minutes, and gate_result.

Second action: run v1.5.3 document ingest (before exploring any code). A single stdlib-only module in bin/ produces the authoritative documentation record that Phase 1 requirement derivation depends on:

• python -m bin.reference_docs_ingest <target> — walks reference_docs/ in the target repo once. Files under reference_docs/cite/ are hashed and written to quality/formal_docs_manifest.json per schemas.md §4 and the §1.6 manifest wrapper. Files at the top level of reference_docs/ are not written to the manifest but are available as Tier 4 context via bin.reference_docs_ingest.load_tier4_context(<target>), which returns a sorted list of (path, text) tuples. If the ingest command fails (unsupported extension, non-UTF-8 bytes), stop the run and surface the stderr output to the user verbatim — ingest errors are actionable and must be fixed before exploration continues.

No sidecar needed. Folder placement is the flag: top-level reference_docs/<name>.<ext> files are Tier 4 context; files under reference_docs/cite/<name>.<ext> are citable sources. Tier 1 is the default for cite/ contents; a file may override to Tier 2 with an optional in-file marker on the first non-blank line: <!-- qpb-tier: 2 --> (Markdown) or # qpb-tier: 2 (plaintext). README.md under either folder is skipped.

When reference_docs/ is missing or empty, Phase 1 MUST print this actionable message and proceed:

Phase 1 found no documentation in reference_docs/. The playbook will proceed using only Tier 3 evidence (the source tree itself). For better results, drop plaintext documentation into: reference_docs/ ← AI chats, design notes, retrospectives (Tier 4 context) reference_docs/cite/ ← project specs, RFCs, API contracts (citable, byte-verified) See README.md "Step 1: Provide documentation" for details.

Plaintext only — conversion happens outside the playbook. Reference docs are .txt or .md only (schemas.md §2). PDFs, DOCX, HTML, etc. are rejected with an actionable conversion hint (pdftotext, pandoc -t plain, lynx -dump). Do NOT attempt to parse binary or formatted documents inside the skill — run the conversion outside and commit the plaintext.

Spend the first phase understanding the project. The quality playbook must be grounded in this specific codebase — not generic advice.

Why explore first? The most common failure in AI-generated quality playbooks is producing generic content — coverage targets that could apply to any project, scenarios that describe theoretical failures, tests that exercise language builtins instead