filtering-bot-traffic

PostHog classifies every request by user agent so you can tell humans apart from bots, crawlers, and AI agents anywhere HogQL runs — the SQL editor, insights, trends, and Web analytics breakdowns. This skill teaches you (the agent) how to use that classification to:

• exclude bots so analytics reflect human traffic only
• measure how much traffic is automated, and which bots / operators are responsible
• separate AI-agent traffic (worth measuring) from noise (worth dropping)
• pick the right surface — virtual properties for the insight builder, functions for raw SQL

For real-time ("right now", last 30 min) bot questions and the Live tab tiles, use the exploring-live-traffic skill instead. This skill is for historical windows, saved insights, dashboards, and filtering.

Use it when the user wants to:

• exclude or filter out bots ("remove bots from my pageviews", "humans only")
• quantify automated traffic ("what % of traffic is bots?", "how much is AI crawlers?")
• find which bots hit them ("which crawlers visit us?", "is ChatGPT reading our docs?")
• break a trend down by traffic type or bot name
• measure AI-agent / AI-search traffic specifically (AEO / answer-engine visibility)

Do not use it for the Live tab, real-time numbers, or the per-minute bot charts — that is exploring-live-traffic.

Two equivalent ways to reach the same classification. Prefer virtual properties in the insight builder and filters; use functions in hand-written SQL or when you need a value the virtual properties don't expose.

getTrafficType / $virt_traffic_type sorts every request into four buckets. The default move differs per bucket — don't treat them all as noise:

The built-in list only covers self-declared user agents PostHog already knows. When a scraper matters to a project but isn't detected — an internal load test, a partner integration, a niche crawler — add a custom bot rule instead of waiting for the built-in list to catch up. Rules extend the same classification surface, so Is bot (isLikelyBot), Bot name, and Traffic category reflect them everywhere HogQL runs.

A rule matches one event property — the user agent by default, but also $ip, $lib, $host, $pathname, $current_url, $browser, $os, $browser_language, $screen_width, $screen_height, $geoip_country_code, $referrer, or $referring_domain — using contains (case-insensitive substring), regex (RE2), or cidr (an IP range, only valid with $ip). Set name to the label reported by Bot name, and optionally category to a built-in category like ai_crawler to relabel the traffic type.

Three ways to manage rules:

• Settings UI — Settings → Environment → Custom bots.
• MCP tools — web-analytics-bot-rules-list, web-analytics-bot-rules-create, web-analytics-bot-rules-destroy. Prefer these when driving PostHog through an agent.
• REST API — GET/POST /api/projects/{project_id}/web_analytics_bot_rules/ and DELETE /api/projects/{project_id}/web_analytics_bot_rules/{id}/.

Listing is open to project members; creating and deleting require a project admin (they mutate the admin-only modifiers team setting). A rule whose pattern can't run is rejected on save, so a bad rule can never take down the project's classification queries.

Most crawlers and AI agents never execute JS, so posthog-js never fires a $pageview for them — they're invisible to client-side analytics. To measure them, the project must forward server access logs as $http_log events carrying $raw_user_agent. If a user asks "why don't I see GPTBot when I know it's crawling us?", the answer is almost always: no $http_log ingestion. Point them at server-side capture (the Vercel logs source, an edge worker, or the capture API) before building bot insights.

• Needs a captured user agent. Classification is computed at query time from the event's $raw_user_agent / $user_agent, so it works on any historical event — there's no need to restrict dateRange.date_from. The one requirement is that a user agent was captured; events from sources that never set one can't be classified (and empty UAs fall through to Automation / no_user_agent, below).
• isLikelyBot is "likely". Detection is a user-agent heuristic — some bots spoof real browser UAs, and some legit tools use bot-like ones. Treat it as best-effort, not ground truth.
• Empty user agent = bot. Requests with no UA (server-to-server, misconfigured SDKs) classify as Automation / no_user_agent, so isLikelyBot returns true.
• Don't silently drop the host filter. If the user is scoped to one domain, inherit $host in properties — leaving it out changes the answer.
• Bot definitions evolve. The detected-bot list changes over time, so re-running the same query later can classify older events differently.