exploring-llm-evaluations

PostHog evaluations score $ai_generation events. Each evaluation is one of three types:

• hog — deterministic Hog code that returns true/false (and optionally N/A). Best for objective rule-based checks: format validation (JSON parses, schema matches), length limits, keyword presence/absence, regex patterns, structural assertions, latency thresholds, cost guards. Cheap, fast, reproducible — no LLM call per run. Prefer this when the criterion can be expressed as code.
• llm_judge — an LLM scores generations against a prompt you write. Best for subjective or fuzzy checks: tone, helpfulness, hallucination detection, off-topic drift, instruction-following. Costs an LLM call per run and requires AI data processing approval at the org level.
• sentiment — classifies sentiment from user messages on each matching generation. Returns a sentiment label and score, not a pass/fail verdict.

Results from all types land in ClickHouse as $ai_evaluation events. Boolean evaluations (llm_judge and hog) set $ai_evaluation_result; sentiment evaluations set $ai_sentiment_* properties instead.

This skill covers the full lifecycle: list/inspect/manage evaluation configs, run them on specific generations, query individual results, and configure evaluation reports that summarize recent runs on a schedule.

All llma-evaluation-* tools are defined in products/ai_observability/mcp/tools.yaml.

Every run of an evaluation emits an $ai_evaluation event. Key properties:

When $ai_evaluation_applicable = false, the run counts as N/A regardless of $ai_evaluation_result. For evaluations that don't support N/A, this property may be null — treat null as "applicable". For boolean evaluations, output_config.true_is_failure: false maps true to pass and false to fail. Set it to true for detector-style evaluations where true means the evaluator found a problem.

Works the same way for boolean llm_judge and hog evaluations — the differences only matter when you eventually go to fix the evaluator (edit the prompt vs. edit the Hog source). Sentiment evaluations should be inspected by sentiment label and score rather than pass/fail filters.

Use this when the user pastes a trace/generation URL and asks "what would evaluation X say about this?".

posthog:llma-evaluation-run
{
  "evaluationId": "<eval_uuid>",
  "target_event_id": "<generation_event_uuid>",
  "timestamp": "2026-04-01T19:39:20Z",
  "event": "$ai_generation"
}

The timestamp is required for an efficient ClickHouse lookup of the target event. Pass distinct_id if you have it — it speeds up the lookup further.

llm_judge evaluations require AI data processing approval at the org level (is_ai_data_processing_approved). Hog evaluations do not require this gate — they run as plain code on the ingestion pipeline.

Reach for Hog by default. Switch to LLM judge only when the criterion can't be expressed as code.

A common pattern is to layer them: a Hog evaluator gates obvious format/length violations cheaply, and an LLM-judge evaluator only fires on the generations that pass the Hog gate (via conditions).

Diagnosis works the same way regardless of whether the evaluator is hog or llm_judge — you read the resulting $ai_evaluation events, not the evaluator itself. The fix path differs (edit Hog source vs. edit prompt) but the diagnosis is identical.

• Evaluations list: https://app.posthog.com/ai-evals/evaluations
• Single evaluation: https://app.posthog.com/ai-evals/evaluations/<evaluation_id>
• Underlying generation/trace: see the exploring-llm-traces skill's URL conventions

Always surface the relevant link so the user can verify in the UI.

• Evaluation reports are configured per evaluation with llma-evaluation-report-create: frequency: "scheduled" with an rrule for a daily or weekly cadence, or frequency: "every_n" with a trigger_threshold to fire once that many new results have accumulated. Delivery goes to email or Slack via delivery_targets
• llma-evaluation-report-generate runs a configured report immediately instead of waiting for the next trigger; llma-evaluation-report-run-list returns past runs with the report content they delivered
• For rich filtering not supported by llma-evaluation-list (e.g. by author or model configuration), fall back to execute-sql against the evaluations Postgres table or the $ai_evaluation ClickHouse events
• When showing failure patterns to the user, always include 1-2 example trace links so they can validate the pattern visually
• llma-evaluation-* tools use evaluation:read for read tools and evaluation:write for mutating tools; the llma-evaluation-report-* tools use llm_analytics:read and llm_analytics:write
• Hog evaluators are reproducible — if you suspect a regression, llma-evaluation-test-hog with the suspect source against the failing generations is the fastest way to bisect whether the change is in the evaluator or in the producer of the generations
• LLM-judge evaluators are non-deterministic across reruns; expect 1-5% noise even with a fixed prompt and model. If you're chasing a small regression in fail rate, prefer Hog or pin a deterministic provider/seed in the model_configuration

• creating-online-evaluations — author a new evaluation from scratch
• exploring-ai-failures — ground the next evaluation in observed failure modes