Labsco
getsentry logo

sentry-setup-ai-monitoring

โœ“ Officialโ˜… 2

by sentry ยท part of getsentry/sentry-for-cursor

Setup Sentry AI Agent Monitoring in any project. Use this when asked to add AI monitoring, track LLM calls, monitor AI agents, or instrument OpenAI/Anthropic/Vercel AI/LangChain/Google GenAI. Automatically detects installed AI SDKs and configures the appropriate Sentry integration.

๐Ÿ”ฅ๐Ÿ”ฅ๐Ÿ”ฅ๐Ÿ”ฅโœ“ VerifiedFreeQuick setup
๐Ÿงฐ Not standalone. This skill ships with getsentry/sentry-for-cursor and only works together with that tool โ€” install the tool first, then add this skill.

This is the playbook your agent receives when the skill activates โ€” you don't need to read it to use the skill, but it's here to audit before installing.

Setup Sentry AI Agent Monitoring

This skill helps configure Sentry's AI Agent Monitoring to track LLM calls, agent executions, tool usage, and token consumption.

When to Use This Skill

Invoke this skill when:

  • User asks to "setup AI monitoring" or "add AI agent tracking"
  • User wants to "monitor LLM calls" or "track OpenAI/Anthropic usage"
  • User requests "AI observability" or "agent monitoring"
  • User mentions tracking token usage, model latency, or AI costs
  • User asks about instrumenting their AI/LLM code with Sentry

CRITICAL: Detection-First Approach

ALWAYS detect installed AI SDKs before suggesting configuration. Do not assume which AI library the user is using.


Step 1: Detect Platform and AI SDKs

For JavaScript/TypeScript Projects

Run these commands to detect installed AI packages:

# Check package.json for AI SDKs
grep -E '"(openai|@anthropic-ai/sdk|ai|@langchain|@google/genai|@langchain/langgraph)"' package.json

Supported JavaScript AI SDKs:

PackageSentry IntegrationMin SDK VersionPackage Version
openaiopenAIIntegration10.2.0>=4.0.0 <7
@anthropic-ai/sdkanthropicAIIntegration10.12.0>=0.19.2 <1
ai (Vercel AI SDK)vercelAIIntegration10.6.0>=3.0.0 <6
@langchain/*langChainIntegration10.22.0>=0.1.0 <1
@google/genaigoogleGenAIIntegration10.14.0>=0.10.0 <2
@langchain/langgraphlangGraphIntegration10.25.0>=0.2.0 <1

For Python Projects

# Check requirements.txt or pyproject.toml
grep -E '(openai|anthropic|langchain|huggingface)' requirements.txt pyproject.toml 2>/dev/null

Supported Python AI SDKs:

PackageSentry ExtraMin SDK VersionPackage Version
openaisentry-sdk[openai]2.41.0>=1.0.0
anthropicsentry-sdk[anthropic]2.x>=0.16.0
langchainsentry-sdk[langchain]2.x>=0.1.11
huggingface_hubsentry-sdk[huggingface_hub]2.x>=0.22.0

Step 2: Verify Sentry SDK Version

JavaScript

grep -E '"@sentry/(nextjs|react|node)"' package.json

Check version meets minimum for detected AI SDK (see table above).

Upgrade if needed:

npm install @sentry/nextjs@latest  # or appropriate package

Python

pip show sentry-sdk | grep Version

Upgrade if needed:

pip install --upgrade "sentry-sdk[openai]"  # include detected extras

Step 3: Verify Tracing is Enabled

AI Agent Monitoring requires tracing. Check that tracesSampleRate is set in Sentry.init().

If not enabled, inform the user:

AI Agent Monitoring requires tracing to be enabled. I'll add tracesSampleRate to your Sentry configuration.

Step 4: Configure Based on Detected SDK

IMPORTANT: Present Detection Results First

Before configuring, tell the user what you found:

I detected the following AI SDK(s) in your project:
- [SDK NAME] (version X.X.X)

Sentry has automatic integration support for this SDK. I'll configure the appropriate integration.

If no supported SDK is detected:

I didn't detect any AI SDKs with automatic Sentry integration support in your project.

Your options:
1. Manual Instrumentation - I can help you set up custom spans for your AI calls
2. Install a supported SDK - If you're planning to add one of the supported SDKs

Would you like to proceed with manual instrumentation? If so, please describe where your AI/LLM calls are located.

Manual Instrumentation (Last Resort)

IMPORTANT: Only use manual instrumentation when:

  1. No supported AI SDK is detected
  2. User explicitly confirms they want manual instrumentation
  3. User describes where their AI calls are located

Ask User Before Proceeding

I'll help you set up manual AI instrumentation. To do this effectively, I need to know:

1. Where are your AI/LLM calls located? (file paths)
2. What AI provider/model are you using?
3. What operations do you want to track?
   - LLM requests (prompts, completions)
   - Agent executions
   - Tool calls
   - Agent handoffs

Manual Span Types

Four span types are required for AI Agents Insights:

1. AI Request Span (gen_ai.request)

Tracks individual LLM calls:

import * as Sentry from "@sentry/nextjs";

async function callLLM(prompt, model = "custom-model") {
  return await Sentry.startSpan(
    {
      op: "gen_ai.request",
      name: `LLM request ${model}`,
      attributes: {
        "gen_ai.request.model": model,
        "gen_ai.request.temperature": 0.7,
        "gen_ai.request.max_tokens": 1000,
      },
    },
    async (span) => {
      // Record input messages
      span.setAttribute(
        "gen_ai.request.messages",
        JSON.stringify([{ role: "user", content: prompt }])
      );

      const startTime = performance.now();

      // Your actual LLM call here
      const result = await yourLLMClient.complete(prompt);

      // Record output and metrics
      span.setAttribute("gen_ai.response.text", result.text);
      span.setAttribute("gen_ai.usage.input_tokens", result.inputTokens || 0);
      span.setAttribute("gen_ai.usage.output_tokens", result.outputTokens || 0);

      return result;
    }
  );
}

2. Invoke Agent Span (gen_ai.invoke_agent)

Tracks full agent execution lifecycle:

async function runAgent(task) {
  return await Sentry.startSpan(
    {
      op: "gen_ai.invoke_agent",
      name: "Execute AI Agent",
      attributes: {
        "gen_ai.agent.name": "my-agent",
        "gen_ai.agent.available_tools": JSON.stringify(["search", "calculate"]),
      },
    },
    async (span) => {
      span.setAttribute("gen_ai.agent.input", task);

      // Agent execution logic (may include multiple LLM calls and tool uses)
      const result = await agent.execute(task);

      span.setAttribute("gen_ai.agent.output", JSON.stringify(result));
      span.setAttribute("gen_ai.usage.total_tokens", result.totalTokens || 0);

      return result;
    }
  );
}

3. Execute Tool Span (gen_ai.execute_tool)

Tracks tool/function calls:

async function executeTool(toolName, toolInput) {
  return await Sentry.startSpan(
    {
      op: "gen_ai.execute_tool",
      name: `Tool: ${toolName}`,
      attributes: {
        "gen_ai.tool.name": toolName,
        "gen_ai.tool.description": getToolDescription(toolName),
      },
    },
    async (span) => {
      span.setAttribute("gen_ai.tool.input", JSON.stringify(toolInput));

      const result = await tools[toolName](https://github.com/getsentry/sentry-for-cursor/blob/main/skills/sentry-setup-ai-monitoring/toolInput);

      span.setAttribute("gen_ai.tool.output", JSON.stringify(result));

      return result;
    }
  );
}

4. Handoff Span (gen_ai.handoff)

Tracks agent-to-agent control transitions:

async function handoffToAgent(fromAgent, toAgent, context) {
  return await Sentry.startSpan(
    {
      op: "gen_ai.handoff",
      name: `Handoff: ${fromAgent} -> ${toAgent}`,
      attributes: {
        "gen_ai.handoff.from_agent": fromAgent,
        "gen_ai.handoff.to_agent": toAgent,
      },
    },
    async (span) => {
      span.setAttribute("gen_ai.handoff.context", JSON.stringify(context));

      // Perform handoff
      const result = await agents[toAgent].receive(context);

      return result;
    }
  );
}

Python Manual Instrumentation

import sentry_sdk
import json

def call_llm(prompt, model="custom-model"):
    with sentry_sdk.start_span(
        op="gen_ai.request",
        name=f"LLM request {model}",
    ) as span:
        span.set_data("gen_ai.request.model", model)
        span.set_data("gen_ai.request.messages", json.dumps([
            {"role": "user", "content": prompt}
        ]))

        # Your actual LLM call
        result = your_llm_client.complete(prompt)

        span.set_data("gen_ai.response.text", result.text)
        span.set_data("gen_ai.usage.input_tokens", result.input_tokens)
        span.set_data("gen_ai.usage.output_tokens", result.output_tokens)

        return result

Span Attributes Reference

Required Attributes

AttributeDescription
gen_ai.request.modelAI model identifier (e.g., "gpt-4o", "claude-3")

Token Usage Attributes

AttributeDescription
gen_ai.usage.input_tokensTokens in prompt/input
gen_ai.usage.output_tokensTokens in response/output
gen_ai.usage.total_tokensTotal tokens used

Request Attributes

AttributeDescription
gen_ai.request.messagesJSON string of input messages
gen_ai.request.temperatureTemperature setting
gen_ai.request.max_tokensMax tokens limit
gen_ai.request.top_pTop-p sampling value

Response Attributes

AttributeDescription
gen_ai.response.textGenerated text response
gen_ai.response.tool_callsJSON string of tool calls

Agent Attributes

AttributeDescription
gen_ai.agent.nameAgent identifier
gen_ai.agent.available_toolsJSON array of tool names
gen_ai.agent.inputAgent input/task
gen_ai.agent.outputAgent output/result

Tool Attributes

AttributeDescription
gen_ai.tool.nameTool identifier
gen_ai.tool.descriptionTool description
gen_ai.tool.inputJSON string of tool input
gen_ai.tool.outputJSON string of tool output

Note: All complex data must be JSON stringified. Span attributes only allow primitive types.


Framework-Specific Notes

Next.js

  • Configure in instrumentation-client.ts, sentry.server.config.ts, sentry.edge.config.ts
  • Vercel AI SDK requires explicit edge runtime configuration
  • Server-side LLM calls captured in server config
  • Client-side calls (if any) in client config

Node.js / Express

  • Configure in entry point or dedicated sentry config
  • All integrations work in Node runtime

React (Client-Only)

  • Limited usefulness - most LLM calls should be server-side
  • Consider moving AI calls to API routes

Python (Django/Flask/FastAPI)

  • Configure in settings or app initialization
  • Wrap LLM calls in transactions for proper span hierarchy

Privacy and PII Considerations

Prompts and outputs are considered PII. To capture them:

JavaScript

Sentry.init({
  sendDefaultPii: true,
  // OR configure per-integration:
  integrations: [
    Sentry.openAIIntegration({
      recordInputs: true,
      recordOutputs: true,
    }),
  ],
});

Python

sentry_sdk.init(
    send_default_pii=True,
    # OR configure per-integration:
    integrations=[
        OpenAIIntegration(include_prompts=True),
    ],
)

To exclude prompts/outputs:

  • Set recordInputs: false / recordOutputs: false (JS)
  • Set include_prompts=False (Python)

Verification Steps

After setup, verify AI monitoring is working:

JavaScript

// Trigger a test LLM call
const response = await openai.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content: "Say 'Sentry test successful'" }],
});
console.log("Test complete:", response.choices[0].message.content);

Python

with sentry_sdk.start_transaction(name="AI Test"):
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "Say 'Sentry test'"}]
    )
    print("Test complete:", response.choices[0].message.content)

Check in Sentry:

  1. Go to Traces > AI Spans tab
  2. Look for spans with gen_ai.* operations
  3. Verify token usage and latency are captured

Summary Checklist


## Quick Reference

| AI SDK | JS Integration | Python Extra |
|--------|----------------|--------------|
| OpenAI | `openAIIntegration()` | `sentry-sdk[openai]` |
| Anthropic | `anthropicAIIntegration()` | `sentry-sdk[anthropic]` |
| Vercel AI | `vercelAIIntegration()` | N/A |
| LangChain | `langChainIntegration()` | `sentry-sdk[langchain]` |
| LangGraph | `langGraphIntegration()` | N/A |
| Google GenAI | `googleGenAIIntegration()` | N/A |
| Hugging Face | N/A | `sentry-sdk[huggingface_hub]` |