auditing-experiments-flags

★ 59

Audit PostHog experiments and feature flags for configuration issues, staleness, and best-practice violations. Read when the user asks to audit, health-check,…

🔥🔥🔥✓ VerifiedFreeQuick setup

#analytics #git

🧩 One of 7 skills in the posthog/ai-plugin package — works on its own, and pairs well with its siblings.

This is the playbook your agent receives when the skill activates — you don't need to read it to use the skill, but it's here to audit before installing.

by posthog

Audit PostHog experiments and feature flags for configuration issues, staleness, and best-practice violations. Read when the user asks to audit, health-check,… npx skills add https://github.com/posthog/ai-plugin --skill auditing-experiments-flags Download ZIP GitHub59

Auditing experiments and feature flags

This skill teaches you how to run configuration audits on experiments and feature flags. All checks use the experiment and feature flag read tools (experiment-get, experiment-list, feature-flag-get-definition, feature-flag-get-all) — no SQL queries are needed for Phase 1 checks.

Output format

For each finding, include:

Severity badge: 🔴 CRITICAL, 🟡 WARNING, or 🔵 INFO
Check name: Which check produced this finding
Entity link: Markdown link to the entity
What's wrong: One-sentence description
Action: What to do about it (see remediation actions)

Example:

🟡 WARNING — Flag integration · Experiment: checkout-redesign The linked feature flag is inactive (paused). Traffic is not being split. Action: Re-enable the flag or end the experiment.

Handling unavailable data

Some checks require activity logs (feature-flags-activity-retrieve for flags), which may not be available in every session. If activity log data is unavailable:

Skip checkActivityHistory (experiment check) entirely.
Skip the "toggle instability" and "never activated" sub-checks in flag lifecycle checks.
In your report, note which checks were skipped and why:

Skipped: Activity history checks (activity logs not available via current tools)

Partial failures

If a fetch call fails for some entities:

Continue with the entities you could fetch.
Report which entities could not be assessed and why.
Do not silently omit entities from the audit.

Reference files

Experiment checks — experiment configuration checks
Flag checks — feature flag checks
Finding types — severity and category definitions
Recurring patterns — patterns across multiple findings
Remediation actions — what to do about each finding

Copy & paste — that's it

npx skills add https://github.com/posthog/ai-plugin --skill auditing-experiments-flags

Run this in your project — your agent picks the skill up automatically.

Usage modes

Quick check (single entity)

When the user asks about a specific experiment or flag:

Fetch the entity via experiment-get (experiment ID) or feature-flag-get-definition (numeric flag ID).
Apply the relevant checks from experiment checks or flag checks.
Report findings inline as markdown, grouped by severity (CRITICAL first, then WARNING, then INFO).
Include entity links as [Experiment: name](/experiments/id) or [Flag: key](/feature_flags/id).

Scoped audit (one domain)

When the user asks to audit all experiments or all flags:

Bulk-fetch via experiment-list or feature-flag-get-all.
Run all checks for that domain against each entity.
Group findings by severity, then by entity.
Report as inline markdown.

Full audit (comprehensive)

When the user asks for a comprehensive audit of both experiments and flags:

Fetch all experiments via experiment-list and all flags via feature-flag-get-all.
Run all experiment checks and all flag checks.
Apply recurring patterns to identify patterns across multiple findings.
If there are more than 5 entities with findings, output as a notebook artifact via notebooks-create for easier navigation. Otherwise report inline.