instrument-llm-analytics

Every call an app makes to an AI model gets recorded on the way out — what it cost, how long it took, how many tokens it used — whichever provider or framework the app is built on.

WHEN TO USE

SCOPE

USE FOR

Record calls to OpenAI or Anthropic with their cost, tokens and speed.
Add the same recording to an app built on LangChain or LlamaIndex.
Tie a recorded AI call back to the person who triggered it.
Wire up an AI provider PostHog has no ready-made integration for.

This is a saved copy of the skill's SKILL.md instructions. It may differ from the upstream file or the version you install.

In this document (4)

Add PostHog LLM analytics

Use this skill to add PostHog LLM analytics that trace AI model usage in new or changed code. Use it after implementing LLM features or reviewing PRs to ensure all generations are captured with token counts, latency, and costs. If PostHog is not yet installed, this skill also covers initial SDK setup. Supports any provider or framework.

Supported providers: OpenAI, Azure OpenAI, Anthropic, Google, Cohere, Mistral, Perplexity, DeepSeek, Groq, Together AI, Fireworks AI, xAI, Cerebras, Hugging Face, Ollama, OpenRouter.

Supported frameworks: LangChain, LlamaIndex, CrewAI, AutoGen, DSPy, LangGraph, Pydantic AI, Vercel AI, LiteLLM, Instructor, Semantic Kernel, Mirascope, Mastra, SmolAgents, OpenAI Agents.

Proxy/gateway: Portkey, Helicone.

Instructions

Follow these steps IN ORDER:

STEP 1: Analyze the codebase and detect the LLM stack.

Look for LLM provider SDKs (openai, anthropic, google-generativeai, etc.) and AI frameworks (langchain, llamaindex, crewai, etc.) in dependency files and imports.
Look for lockfiles to determine the package manager.
Check for existing PostHog or observability setup. If PostHog is already installed and LLM tracing is configured, skip to STEP 4 to add tracing for any new LLM calls.

STEP 2: Research instrumentation. (Skip if PostHog LLM tracing is already set up.) 2.1. Find the reference file below that matches the detected provider or framework — it is the source of truth for callback setup, middleware configuration, and event capture. Read it now. 2.2. If no reference matches, use manual-capture.md as a fallback — it covers the generic event capture approach that works with any provider.

STEP 3: Install the PostHog SDK. (Skip if PostHog is already set up.)

Add the PostHog SDK and any required callback/integration packages.
Do not manually edit dependency files — use the package manager's install command.

STEP 4: Add LLM tracing.

Instrument LLM calls to capture input tokens, output tokens, model name, latency, and costs for every generation.
Follow the provider-specific reference for the exact callback/middleware setup.
Do not alter the fundamental architecture of existing files. Make additions minimal and targeted.
You must read a file immediately before attempting to write it.

STEP 5: Link to users.

Associate LLM generations with identified users via distinct IDs when possible.

STEP 6: Set up environment variables.

Check if the project already has PostHog environment variables configured (e.g. in .env, .env.local, or framework-specific env files). If valid values already exist, skip this step.
If the PostHog project token is missing, use the PostHog MCP server's projects-get tool to retrieve the project's api_token. If multiple projects are returned, ask the user which project to use. If the MCP server is not connected or not authenticated, ask the user for their PostHog project token instead.
For the PostHog host URL: check the projects-get MCP response for a region field — US maps to https://us.i.posthog.com, EU maps to https://eu.i.posthog.com. If the region is not available from the MCP response or from existing project configuration, ask the user: "Are you on PostHog US Cloud or EU Cloud?" Do not assume US Cloud.
Write these values to the appropriate env file using the framework's naming convention.
Reference these environment variables in code instead of hardcoding them.

Reference files

references/openai.md - Openai observability installation - docs
references/azure-openai.md - Azure openai observability installation - docs
references/README.md - PostHog.ai for .net
references/anthropic.md - Anthropic ai observability installation - docs
references/google.md - Google ai observability installation - docs
references/cohere.md - Cohere ai observability installation - docs
references/mistral.md - Mistral ai observability installation - docs
references/perplexity.md - Perplexity ai observability installation - docs
references/deepseek.md - Deepseek ai observability installation - docs
references/groq.md - Groq ai observability installation - docs
references/together-ai.md - Together ai observability installation - docs
references/fireworks-ai.md - Fireworks ai observability installation - docs
references/xai.md - Xai observability installation - docs
references/cerebras.md - Cerebras ai observability installation - docs
references/hugging-face.md - Hugging face ai observability installation - docs
references/ollama.md - Ollama ai observability installation - docs
references/openrouter.md - Openrouter ai observability installation - docs
references/langchain.md - Langchain ai observability installation - docs
references/llamaindex.md - Llamaindex ai observability installation - docs
references/crewai.md - Crewai observability installation - docs
references/autogen.md - Autogen ai observability installation - docs
references/dspy.md - Dspy ai observability installation - docs
references/langgraph.md - Langgraph ai observability installation - docs
references/pydantic-ai.md - Pydantic ai observability installation - docs
references/vercel-ai.md - Vercel ai SDK observability installation - docs
references/litellm.md - Litellm ai observability installation - docs
references/instructor.md - Instructor ai observability installation - docs
references/semantic-kernel.md - Semantic kernel ai observability installation - docs
references/mirascope.md - Mirascope ai observability installation - docs
references/mastra.md - Mastra ai observability installation - docs
references/smolagents.md - Smolagents ai observability installation - docs
references/openai-agents.md - Openai agents SDK observability installation - docs
references/portkey.md - Portkey ai observability installation - docs
references/helicone.md - Helicone ai observability installation - docs
references/manual-capture.md - Manual capture ai observability installation - docs
references/basics.md - Ai observability basics - docs
references/traces.md - Traces - docs
references/calculating-costs.md - Calculating llm costs - docs
references/COMMANDMENTS.md - Framework-specific rules the integration must follow

Each provider reference contains installation instructions, SDK setup, and code examples specific to that provider or framework. Find the reference that matches the user's stack.

If the user's provider isn't listed, use manual-capture.md as a fallback — it covers the generic event capture approach that works with any provider.

Key principles

Environment variables: Always use environment variables for PostHog and LLM provider keys. Never hardcode them.
Minimal changes: Add LLM analytics alongside existing LLM calls. Don't replace or restructure existing code.
Trace all generations: Capture input tokens, output tokens, model name, latency, and costs for every LLM call.
Link to users: Associate LLM generations with identified users via distinct IDs when possible.
One provider at a time: Only instrument the provider(s) the user is actually using. Don't add instrumentation for providers not present in the codebase.

This skill ships inside the PostHog AI Plugin. Listed package contents include 3 slash commands, 1 agent, hooks and an MCP server. Installed components and skill versions may vary by AI tool.

Install just this skill

Run this outside your AI tool, using Node.js 22.20.0 or newer. Skills CLI 1.7.2 skips skill selection; prompts cancel installation without an interactive terminal. If choosing Project when asked about installation scope, open your project's folder before running this.

Install checklist

Check selected tools if a picker appears; Claude Code may be unchecked under Additional agents. ↑/↓ moves, Space toggles selections, Enter confirms.
Without a tool picker, check Installing to: for your intended tool. Stop if it names a different tool.
If asked, choose Symlink (Recommended) for shared files or Copy to all agents for separate copies; review the summary before confirming installation.
Install the find-skills skill? defaults to Yes, adding another skill for all projects. Choose No if you don't want that extra installation.

Skills CLI 1.7.2 sends install data to skills.sh by default, including the source, skill names, target tools and file details; GitHub sources are reported only when GitHub confirms the repository is public, while other remote sources can also be reported. Confirmed-public GitHub sources also trigger a security-audit request. Before installing, set the environment variable DO_NOT_TRACK=1 or DISABLE_TELEMETRY=1 to disable both.

npx skills add posthog/ai-plugin --skill "instrument-llm-analytics" --full-depth

After installing for Claude Code, open it in the installation project if you chose Project; Global makes the skill available across projects on this computer. Once Claude Code loads the skill, its description helps Claude decide when to use it.

If installation created a top-level skills folder that did not exist when your Claude Code session started, run /reload-skills in that session. Run it again after later changes in that folder.

Install the whole plugin

This route installs the posthog plugin package selected for your AI tool.

Claude Code

Add PostHog's marketplace, then install the plugin

Run both lines in a terminal. They install PostHog's posthog plugin: this skill, PostHog's other skills, and a connection to PostHog's hosted MCP server.

claude plugin marketplace add PostHog/ai-plugin

02
```
claude plugin install posthog@posthog
```

Installation and loading

The plugin installs for you in every project and loads the next time you start Claude Code, or after /reload-plugins in an open session.
To connect your PostHog account, start claude, type /mcp, select plugin:posthog:posthog and follow the browser prompts.

Good to know

The plugin includes PostHog's session-capture hook; PostHog's README says it sends sessions to LLM Analytics only when POSTHOG_LLMA_CC_ENABLED=true and POSTHOG_API_KEY are both set.
If you use the separate posthog-telemetry plugin, PostHog says to keep only one of the two, because both register the same hook.
On a self-hosted PostHog, this route's MCP connection won't reach your instance: the plugin's config is fixed to https://mcp.posthog.com/mcp, which routes to PostHog's US or EU region. POSTHOG_MCP_URL works only for the Gemini CLI extension.

What's inside

Files included with this skill.

View on GitHub

Same repository

adding-warehouse-person-properties
Sync columns from a synced data warehouse table onto PostHog person or group properties, so warehouse data becomes usable anywhere person and group properties already work: feature flag targeting, cohorts, insight filters and breakdowns, surveys, session replay filters, workflows, and the person profile. Use when the user wants to "add a person property from my warehouse", "enrich people with Stripe/Postgres/Salesforce data", "put ARR or plan tier on my persons", "target a feature flag by a warehouse column", "sync warehouse columns onto groups or organizations", or wants to inspect, backfill, disable, or debug an existing warehouse-backed person or group property.Read full description
analyzing-expensive-users
Analyze the most expensive users in AI observability and explain why they cost so much. Use when the user asks about top spenders, expensive users, per-user LLM cost, user-level cost drivers, or patterns behind high AI observability spend.Read full description
analyzing-experiment-session-replays
Analyze session replay patterns across experiment variants to understand user behavior differences. Use when the user wants to see how users interact with different experiment variants, identify usability issues, compare behavior patterns between control and test groups, or get qualitative insights to complement quantitative experiment results. Also covers pairing the observed behavior with a linked survey when the user wants qualitative feedback beyond what recordings show.Read full description
assessing-heatmaps
Assesses what a page's heatmap is telling you and recommends concrete changes. Pulls click / rageclick / scroll-depth data for a URL, names the hot elements by cross-referencing autocapture events on the same page, and can create a saved heatmap the user opens in PostHog, then summarizes the behavior and proposes improvements. TRIGGER when: user asks what a heatmap shows, why people aren't clicking something, where users rage-click, how far they scroll, what to change on a page based on heatmap/click data, or to 'analyze/assess/review the heatmap' for a URL. DO NOT TRIGGER when: the user only wants to create a saved heatmap screenshot with no analysis (use heatmaps-saved-create directly), or is asking about session replay in general (use investigating-replay).Read full description
auditing-endpoints
Audit every endpoint in a PostHog project for staleness, failed materialisations, and unused materialised versions. Use when the user asks "what endpoints can I clean up?", "are any of my endpoints broken?", "which materialised versions are still being called?", or wants a one-shot cleanup pass over the Endpoints product. Produces a prioritised report grouped by issue type, with recommended actions but does not modify anything without explicit confirmation.Read full description
auditing-experiments-flags
Audit PostHog experiments and feature flags for configuration issues, staleness, and best-practice violations. Read when the user asks to audit, health-check, or review experiments or feature flags, check flag hygiene, or verify experiment setup.Read full description