skill-progressive-disclosure-design

User intent selects exactly one path. SKILL.md holds the decision logic and shared workflow. Each references/<variant>.md holds path-specific detail.

my-skill/
├── SKILL.md               # decision tree + shared steps
└── references/
    ├── variant-a.md
    ├── variant-b.md
    └── variant-c.md

Examples of clean variants: cloud provider, database engine, framework choice, output format, language.

Pros:

• Each invocation loads only the matching variant; large savings when variants are big.
• Variants evolve independently, simplifying maintenance.
• Adding a new variant does not bloat existing content.
• Mental model is easy: select one path based on input.
• Maps cleanly to user intent that already mentions the variant.

Cons:

• Requires routing logic in SKILL.md, eating back some of the line savings.
• Cross-cutting changes touch every variant file, multiplying effort.
• Risk of treatments diverging across variants over time.
• If user intent is ambiguous, the model may load multiple variants and lose the savings.
• If variants share more than ~60% of their content, the abstraction breaks down.

SKILL.md holds the procedure (verbs, sequence, decisions). references/ holds lookup material queried by key.

Good reference content: schemas, error code tables, API surface listings, example galleries, configuration option matrices, design tokens.

Pros:

• Highest leverage of all splits: lookups are narrow, the model reads one entry.
• Natural conceptual boundary (procedure vs. data).
• Reference can grow large without affecting per-invocation cost.
• Adding new reference entries does not touch the workflow.
• Reference data can often be machine-generated and regenerated.

Cons:

• The model must know what to look up before reading. Pointer must encode lookup keys explicitly.
• Fails when the workflow needs to weave reference data inline rather than at discrete points.
• Splits content that is conceptually unified, harder for human readers.
• The model may miss broader context that lives only in the reference.
• Lookup data that is small (under ~50 lines total) is rarely worth splitting.

SKILL.md covers the 80% case. references/edge-cases.md covers the rest.

The pointer must read like:

If you see X, Y, or Z, stop and read references/edge-cases.md before continuing.

Pros:

• Common path stays minimal, fast, cheap.
• Edge cases can be exhaustive without polluting every invocation.
• Easy to extend edge-case coverage without touching the common path.
• Mirrors how experts work: defaults first, exceptions on demand.

Cons:

• The load condition must be sharp and observable from user input. Most edge cases do not satisfy this.
• Vague conditions cause either always-loading (waste) or never-loading (dead weight).
• Edge cases get less testing because evals naturally cluster on common queries.
• The model may follow the common path past a point where it should have escalated.
• The 80/20 estimate is often wrong; what looked like an edge case turns out to be common.

A testing skill split into unit.md, integration.md, mocks.md is a typical example.

Why it appears attractive:

• Conceptually clean, mirrors how a human would organize documentation.
• Easy to navigate as a maintainer.
• Plausibly reduces context per invocation.

Why it fails:

• Real tasks span 2-3 topics, forcing multiple loads per invocation.
• Cross-topic concerns get duplicated or fragmented.
• The savings are theoretical, not empirical.

Why it appears attractive:

• A heuristic ("keep SKILL.md under 400 lines") feels like a clean rule to satisfy.
• Splitting feels like progress.

Why it fails:

• Without a branching condition, references load in parallel or always, providing no savings.
• Adds router prose to SKILL.md, often making the total content longer.

Why it appears attractive:

• The content is large or specialized.
• Moving it out of SKILL.md feels like good hygiene.

Why it fails:

• References are optional by design; the model may skip them.
• If the content is critical, it must be loaded reliably, which means SKILL.md.
• "Rare" and "critical" together is usually a sign the skill is doing two jobs and should be two skills.

Why it appears attractive:

• Reduces visual clutter in SKILL.md.
• Feels like good organization.

Why it fails:

• No load condition: either always loaded (wasted tool call) or never loaded (dead content).
• Implies an importance hierarchy that does not exist at runtime.
• Frequently hides content from the model that needs it.

Output evals optimize for output quality across realistic queries. Architecture evals optimize for path coverage. The eval set must exercise every code path the skill claims to have, otherwise the metrics are noise.

Construct, at minimum:

• One query per declared variant (if the skill uses variant-branch splits).
• One query per edge-case branch (if depth-tier splits exist).
• One query per major lookup category (if reference-data splits exist).
• One query that should hit the common path only and load zero references.
• 2-3 off-topic queries that should not trigger the skill at all (also tests the description).

If no realistic query triggers a given reference file, that file is dead. Inline it or delete it before running anything.

Each eval run is executed by a subagent with the skill loaded. Capture per run:

1. Full transcript including every tool call.
2. Which references/* files were read (parse view calls on paths inside the skill directory).
3. Whether scripts/* were invoked.
4. Total tokens and wall time.
5. The output (for the parallel output-quality eval).

Persist as transcript.json and loads.json per run, alongside the standard output. The harness from skill-creator already records tokens and time in timing.json; extend its grading step to extract reference loads from transcripts.

Across all eval runs, for each references/*.md:

• Load rate: fraction of runs that read it.
• Co-occurrence: for each other reference, fraction of runs that loaded both.
• Use rate when loaded: of the runs that loaded it, did the content visibly inform the output (cited content, applied procedure, used schema)? Inspect transcripts.
• Re-read rate: fraction of runs that loaded the same file twice.

• Median and p95 tokens per invocation, with and without references.
• SKILL.md utilization: read transcripts and identify sections of SKILL.md the model never references in any run. Strong candidates for deletion.
• Path coverage: did every declared path get hit by at least one query?

These thresholds are starting points. Tune them based on the cost profile: small references with cheap loads tolerate lower load rates than large ones.

When choosing between architectures (monolithic vs. split, or split A vs. split B):

1. Run the identical eval set against both versions.
2. Run output-quality evals on both. Confirm no regression. If quality drops, the architecture change is a loss regardless of token savings.
3. Compare median tokens, p95 tokens, and median time per run.
4. Compare path coverage: does each version reliably reach the same outputs through the expected paths?

A split that saves 15% tokens but adds variance in output quality is worse than the monolith. Reliability beats efficiency.

• Whether the SKILL.md prose is clear. Read transcripts for confused tool calls and dead-end attempts.
• Whether the description triggers correctly. That is a separate eval (use run_loop.py from the skill-creator skill).
• Whether content placement matches user mental models. Subjective; review with a human.

The split that looked clean at design time rarely matches real load patterns. Trust the transcripts over your intuitions.