creating-online-evaluations

Let the data decide how many. One failure mode is one eval, so a ranked list of four distinct modes is a candidate set of four evals. Don't collapse them into one evaluator that tries to catch everything, and don't stop at the top mode when the traces clearly showed more worth watching. Keep a candidate when:

• It hurts. Frequent or painful. A handful of modes usually covers the majority of failures.
• It's checkable. It reduces to one crisp criterion — "the reply must stay on the user's topic", "the tool call must include an order_id". If you can't state it in a line, it isn't ready to propose.
• It's distinct. Two candidates that would fail on the same generations are one eval.

Rank by how much they hurt and propose roughly the top five; mention in a line that you set weaker ones aside rather than silently dropping them.

Never create evals the user hasn't picked. Lay out the candidates and ask which ones they want.

Assume they haven't read the traces with you and don't know the eval vocabulary. Keep each one to a line or two — a wall of text per eval means they can't compare them — but make it obvious what it watches and what they'll see when it fails. No Hog snippets, property filters, or hog/llm_judge internals here. Number them so they can reply "1 and 3":

Found 3 failure patterns worth watching. Which should I set up?

1. Replies drift off topic — checks the answer addresses what the user actually asked. Catches support replies that confidently answer a different question. Seen ~40×/day.

2. Order lookups missing an order ID — checks every lookup_order call includes an order_id. Catches the silent tool failures that make the agent invent an order status. Seen ~15×/day.

3. Refuses questions it can answer — checks the reply isn't a needless "I can't help with that". Catches users being turned away from things the docs cover. Costs an LLM call per check.

Per eval: what it checks, what bad thing it surfaces, and rough volume when you know it (Phase 2.5 verifies it properly). Flag the ones needing an LLM judge, since those cost per run — that's the only mechanic worth exposing up front. Keep the ranking implicit in the order; skip scoring tables.

Recommend a starting point if they seem unsure (usually the top one or two), and treat "all of them" or "go ahead" as accepting the whole set. If a prompt tweak would likely fix a mode, suggest the fix alongside the eval rather than in place of it — a rising pass rate is how they confirm the fix landed.

If the traces were read and no failure mode is worth an eval, don't invent a failure and don't come back empty-handed. Say what you looked at, then propose a sentiment eval as the floor:

No clear failure pattern in the last 7 days. Worth starting with:

1. User frustration — labels each user message positive, neutral, or negative. Shows which conversations are going badly, which is usually where the real failures hide. No judge cost.

It needs no provider key, it's cheap, and it gives them a signal to come back to once there's enough traffic to spot patterns.

No generations means no eval of any kind. Sentiment only scores matching $ai_generation events, so if the project has none, every eval you could create — sentiment included — would sit there never firing. Don't propose one. Point them at instrument-llm-analytics to get AI observability capturing generations first, and come back to this skill once there's traffic to read.

Reach for hog first, escalate to llm_judge if there is no deterministic way to check for what we want to check.

For a trace target, send "target": "trace" plus a settle config that controls when the trace is evaluated, discriminated on strategy:

• { "strategy": "fixed_window", "window_seconds": 1800 } — evaluate a fixed wait after the first matching generation. Between 10 seconds and 2 hours, defaults to 30 minutes. A target_config without a strategy key means this.
• { "strategy": "inactivity", "quiet_period_seconds": 300, "max_age_seconds": 7200 } — evaluate once the trace has had no new activity for the quiet period (10 seconds to 30 minutes, defaults to 5 minutes). max_age_seconds caps the total wait from the first matching generation (1 minute to 2 hours, defaults to 2 hours, must be at least the quiet period).

A session target takes the same settle config with session-sized bounds, and defaults to inactivity rather than fixed_window:

• { "strategy": "inactivity", "quiet_period_seconds": 3600, "max_age_seconds": 86400 } — evaluate once the session has had no new activity for the quiet period (10 seconds to 24 hours, defaults to 1 hour). max_age_seconds caps the total wait from the first matching generation (1 minute to 7 days, defaults to 24 hours, must be at least the quiet period).
• { "strategy": "fixed_window", "window_seconds": 1800 } — evaluate a fixed wait after the first matching generation (10 seconds to 7 days).

A session evaluation only fires for events that carry $ai_session_id. Producers either set it on every generation or on none, so an SDK that does not set it will never trigger a session evaluation. $ai_session_id is not $session_id: the second is PostHog's product-analytics session and is unrelated.

A session evaluation can also come back skipped rather than graded. The emitted $ai_evaluation event then carries $ai_evaluation_skipped: true and an $ai_evaluation_skip_reason, and its $ai_evaluation_result is false when the evaluation disallows N/A, so any analysis of pass rates has to exclude skipped runs rather than count them as failures. Sessions are skipped when they hold more than 2500 events (usually a session id shared across conversations), when nothing was found in the evaluation window, and, for an LLM judge, when the transcript is too long to send in full.

A session is evaluated at most once per evaluation, for as long as the completed run stays inside Temporal's retention window. A session that resumes long after being evaluated may be evaluated again, so pick a quiet period long enough that the session is really finished. A longer quiet period costs only latency.

Conditions still match the generation that triggers the run; the evaluator itself receives the complete trace or session. Sentiment evaluations support only the generation target.

New Hog source should use the globals shared by all targets:

For a session target, target.id is the session id, and target.total_cost_usd / target.total_latency_seconds are summed across the session's traces. total_latency_seconds is time spent on AI work, not session wall-clock; the two can differ by orders of magnitude. Session wall-clock is derivable from evaluation_events timestamps.

Generation evaluations still expose top-level input, output, properties, and event. Trace evaluations still expose their original events and trace globals. Those globals are kept for compatibility with saved evaluators. Session evaluations do not carry them: session Hog source only receives target and evaluation_events. Do not use target-specific globals in new source that needs to work across targets. The text projections recognize common provider payloads but are not authoritative; use item.input / item.output when exact structure matters.

An llm_judge evaluation requires only a valid provider and model to create as a draft. A usable provider key is needed to enable it, not to save it, so no ok key is a reason to keep the eval disabled, not a reason to stop. provider_key_id controls whether the evaluation pins one specific key:

• Set provider_key_id to the UUID of an ok-state key for the same provider to pin it.
• Set provider_key_id to null to use the team's active provider key. This is always a valid saved value. To enable the eval, the active key must be in the ok state and use the same provider as model_configuration.provider.

Hog and sentiment evaluations skip this step.

posthog:llma-evaluation-config-get        // check active_provider_key for an unpinned judge
posthog:llma-provider-key-list            // find an ok-state key to pin
posthog:llma-evaluation-judge-models      // {} → every provider and its models; { "provider": "openai" } narrows it

Confirm the provider and model with llma-evaluation-judge-models. Call it with no arguments to see the whole catalog at once. Providers PostHog funds no models for come back empty unless you pass key_id for one of the team's keys; the response's providers list flags which ones those are. Prefer pinning the chosen key so a later team-wide active-key change does not change how the evaluation runs. Leave provider_key_id as null to run on the team's active key; before enabling, confirm with llma-evaluation-config-get that the active key is usable and its provider matches.

No usable key does not block creation. Create the disabled draft with a valid provider and model for the user to review. Do not spot-run or enable it, and ask the user to add or validate a key in the UI so it can be enabled later.

Create with enabled: false so nothing fires until the scope is verified. Minimal hog example:

Evaluations may be created at the top level or in one directory. If the user names a directory, call posthog:llma-evaluation-directory-list and pass its UUID as directory_id. Create a directory only when the user asks for one. Omit directory_id or pass null for the top level. Directories cannot be nested.

posthog:llma-evaluation-create
{
  "name": "Output is not empty",
  "description": "Fails when a generation has no readable output",
  "evaluation_type": "hog",
  "evaluation_config": { "source": "let count := 0\nfor (let i, item in evaluation_events) {\n    if (item.event == '$ai_generation') {\n        count := count + 1\n        if (length(trim(item.output_text)) == 0) { return false }\n    }\n}\nreturn count > 0" },
  "output_type": "boolean",
  "output_config": { "allows_na": false },
  "target": "generation",
  "target_config": {},
  "conditions": [
    { "id": "default", "rollout_percentage": 100, "properties": [{ "key": "$ai_model", "type": "event", "operator": "icontains", "value": "gpt" }] }
  ],
  "enabled": false
}

For llm_judge, swap evaluation_config to { "prompt": "…" } and add "model_configuration": { "provider": "openai", "model": "gpt-5-mini", "provider_key_id": null }. Pin provider_key_id to the UUID of an ok-state key from llma-provider-key-list to run on one specific key; null runs on the team's active key. Either value saves a draft. Full field reference: references/evaluation-payload.md.

conditions is where online evals go wrong: too broad and you evaluate (and bill) a firehose; too narrow and it never fires. Confirm the filter matches the events you expect, and roughly how many per day:

posthog:execute-sql
SELECT count() AS matched, count() / 7 AS per_day
FROM events
WHERE event = '$ai_generation'
    AND properties.$ai_model ILIKE '%gpt%'      -- mirror each condition property
    AND timestamp >= now() - INTERVAL 7 DAY

For generation targets, count() is the run volume. For trace targets, count distinct non-empty $ai_trace_id values because matching generations from the same trace schedule only one run.

If volume is high, set rollout_percentage below 100 to sample. Spot-check the evaluator with llma-evaluation-test-hog (hog) or llma-evaluation-run against one generation (llm_judge; skip the spot-run for a judge with no usable key, per 2.3). Both tools currently use generation samples; for a trace target they can check shared source or prompt behavior, but they do not reproduce the complete settled trace. Review the first live trace results before increasing rollout.

Watch out: some orgs reuse a single $ai_trace_id across 100k+ events. Scoping by trace-ID prefix can match far more than expected — verify volume with the SQL above before enabling.

Only once every accepted eval is a scope-verified draft, enable them — one call each:

posthog:llma-evaluation-update
{ "evaluationId": "<uuid>", "enabled": true }

Each now runs on every new matching generation, or once per matching trace for a trace target. This isn't one-and-done: the user should be aware that they need to keep an eye on results and iterate if the outcome is not the expected one. To wire results into a Slack feed, see feature-usage-feed.

Close the loop across the whole set at once — one short list of what's now live with a link each, not a play-by-play per eval. Mention any candidate you left disabled (no usable provider key, volume too high to enable yet) and what would unblock it.

An enabled eval only scores traffic from now on. If the user wants results on data they already have, offer a backfill over a recent window, at most the last 30 days.

1. Call posthog:llma-evaluation-backfill-estimate with the window, conditions, and rerun_existing the user wants. Tell the user total_units, the returned window (it is clamped, so it can differ from the one you asked for), and what each unit costs: one run of the eval, counted as an AI observability event, plus a model call for an llm_judge. already_evaluated_units already have a result and are left out unless rerun_existing is true.
2. Call posthog:llma-evaluation-backfill-create only after the user says yes. Pass the window_start and window_end the estimate returned, not the ones you asked for, with the same conditions and rerun_existing, so the run matches what they approved.
3. Track it with posthog:llma-evaluation-backfill-get. Skipped units were already being evaluated by the eval itself, so they are covered, not missed. Once it completes, a remaining_count above zero means units were left without a result; offer another backfill over the same range. Offer it once. If the second run leaves units behind again, the eval keeps failing on them, so tell the user instead of offering a third.
4. If posthog:llma-evaluation-backfill-create says it could not confirm the start, call posthog:llma-evaluation-backfill-list before trying again, because the backfill may be running.
5. To stop a backfill, call posthog:llma-evaluation-backfill-cancel. Evaluations it already started still finish.

One backfill runs per evaluation at a time.