langsmith-trace

Nothing can be debugged or turned into test cases unless the app records what it actually did; this puts that recording in place, and then searches it.

WHEN TO USE

SCOPE

USE FOR

Turn on automatic tracing for a LangChain or LangGraph application.
Add tracing to a non-LangChain app with the traceable decorator.
Find the traces that ran slowest or failed in the last hour.
Export a batch of traces to JSONL files to build an evaluation dataset.
Filter traces by a specific feedback score to surface weak responses.

This is a saved copy of the skill's SKILL.md instructions. It may differ from the upstream file or the version you install.

Two main topics: adding tracing to your application, and querying traces for debugging and analysis. Python and Javascript implementations are both supported.

Environment Variables

LANGSMITH_API_KEY=lsv2_pt_your_api_key_here          # REQUIRED
LANGSMITH_PROJECT=your-project-name                   # Optional: default project
LANGSMITH_WORKSPACE_ID=your-workspace-id              # Optional: for org-scoped keys

Authentication is REQUIRED: either set the LANGSMITH_API_KEY environment variable, or pass the --api-key flag to CLI commands (preferred):

langsmith trace list --project my-project --api-key $LANGSMITH_API_KEY

IMPORTANT: Always check the environment variables or .env file for LANGSMITH_PROJECT before querying or interacting with LangSmith. This tells you which project contains the relevant traces and data. If the LangSmith project is not available, use your best judgement to identify the right one.

CLI Tool

curl -sSL https://raw.githubusercontent.com/langchain-ai/langsmith-cli/main/scripts/install.sh | sh

<trace_langchain_oss> For LangChain/LangGraph apps, tracing is automatic. Just set environment variables:

export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY=<your-api-key>
export OPENAI_API_KEY=<your-openai-api-key>  # or your LLM provider's key

Optional variables:

LANGSMITH_PROJECT - specify project name (defaults to "default")
LANGCHAIN_CALLBACKS_BACKGROUND=false - use for serverless to ensure traces complete before function exit (Python) </trace_langchain_oss>

<trace_other_frameworks> For non-LangChain apps, if the framework has native OpenTelemetry support, use LangSmith's OpenTelemetry integration.

If the app is NOT using a framework, or using one without automatic OTel support, use the traceable decorator/wrapper and wrap your LLM client.

Use @traceable decorator and wrap_openai() for automatic tracing.

from langsmith import traceable
from langsmith.wrappers import wrap_openai
from openai import OpenAI

client = wrap_openai(OpenAI())

@traceable
def my_llm_pipeline(question: str) -> str:
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": question}],
    )
    return resp.choices[0].message.content

# Nested tracing example
@traceable
def rag_pipeline(question: str) -> str:
    docs = retrieve_docs(question)
    return generate_answer(question, docs)

@traceable(name="retrieve_docs")
def retrieve_docs(query: str) -> list[str]:
    return docs

@traceable(name="generate_answer")
def generate_answer(question: str, docs: list[str]) -> str:
    return client.chat.completions.create(...)

Use traceable() wrapper and wrapOpenAI() for automatic tracing.

import { traceable } from "langsmith/traceable";
import { wrapOpenAI } from "langsmith/wrappers";
import OpenAI from "openai";

const client = wrapOpenAI(new OpenAI());

const myLlmPipeline = traceable(async (question: string): Promise<string> => {
  const resp = await client.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [{ role: "user", content: question }],
  });
  return resp.choices[0].message.content || "";
}, { name: "my_llm_pipeline" });

// Nested tracing example
const retrieveDocs = traceable(async (query: string): Promise<string[]> => {
  return docs;
}, { name: "retrieve_docs" });

const generateAnswer = traceable(async (question: string, docs: string[]): Promise<string> => {
  const resp = await client.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [{ role: "user", content: `${question}\nContext: ${docs.join("\n")}` }],
  });
  return resp.choices[0].message.content || "";
}, { name: "generate_answer" });

const ragPipeline = traceable(async (question: string): Promise<string> => {
  const docs = await retrieveDocs(question);
  return await generateAnswer(question, docs);
}, { name: "rag_pipeline" });

Best Practices:

Apply traceable to all nested functions you want visible in LangSmith
Wrapped clients auto-trace all calls — wrap_openai()/wrapOpenAI() records every LLM call
Name your traces for easier filtering
Add metadata for searchability </trace_other_frameworks>

<traces_vs_runs> Use the langsmith CLI to query trace data.

Understanding the difference is critical:

Trace = A complete execution tree (root run + all child runs). A trace represents one full agent invocation with all its LLM calls, tool calls, and nested operations.
Run = A single node in the tree (one LLM call, one tool call, etc.)

Generally, query traces first — they provide complete context and preserve hierarchy needed for trajectory analysis and dataset generation. </traces_vs_runs>

<command_structure> Two command groups with consistent behavior:

langsmith
├── trace (operations on trace trees - USE THIS FIRST)
│   ├── list    - List traces (filters apply to root run)
│   ├── get     - Get single trace with full hierarchy
│   └── export  - Export traces to JSONL files (one file per trace)
│
├── run (operations on individual runs - for specific analysis)
│   ├── list    - List runs (flat, filters apply to any run)
│   ├── get     - Get single run
│   └── export  - Export runs to single JSONL file (flat)
│
├── dataset (dataset operations)
│   ├── list    - List datasets
│   ├── get     - Get dataset details
│   ├── create  - Create empty dataset
│   ├── delete  - Delete dataset
│   ├── export  - Export dataset to file
│   └── upload  - Upload local JSON as dataset
│
├── example (example operations)
│   ├── list    - List examples in a dataset
│   ├── create  - Add example to a dataset
│   └── delete  - Delete an example
│
├── evaluator (evaluator operations)
│   ├── list    - List evaluators
│   ├── upload  - Upload evaluator
│   └── delete  - Delete evaluator
│
├── experiment (experiment operations)
│   ├── list    - List experiments
│   └── get     - Get experiment results
│
├── thread (thread operations)
│   ├── list    - List conversation threads
│   └── get     - Get thread details
│
└── project (project operations)
    └── list    - List tracing projects

Key differences:

	`traces *`	`runs *`
Filters apply to	Root run only	Any matching run
`--run-type`	Not available	Available
Returns	Full hierarchy	Flat list
Export output	Directory (one file/trace)	Single file
</command_structure>

<querying_traces> Query traces using the langsmith CLI. Commands are language-agnostic.

# List recent traces (most common operation)
langsmith trace list --limit 10 --project my-project --api-key $LANGSMITH_API_KEY

# List traces with metadata (timing, tokens, costs)
langsmith trace list --limit 10 --include-metadata --api-key $LANGSMITH_API_KEY

# Filter traces by time
langsmith trace list --last-n-minutes 60 --api-key $LANGSMITH_API_KEY
langsmith trace list --since 2025-01-20T10:00:00Z --api-key $LANGSMITH_API_KEY

# Get specific trace with full hierarchy
langsmith trace get <trace-id> --api-key $LANGSMITH_API_KEY

# List traces and show hierarchy inline
langsmith trace list --limit 5 --show-hierarchy --api-key $LANGSMITH_API_KEY

# Export traces to JSONL (one file per trace, includes all runs)
langsmith trace export ./traces --limit 20 --full --api-key $LANGSMITH_API_KEY

# Filter traces by performance
langsmith trace list --min-latency 5.0 --limit 10 --api-key $LANGSMITH_API_KEY    # Slow traces (>= 5s)
langsmith trace list --error --last-n-minutes 60 --api-key $LANGSMITH_API_KEY     # Failed traces

# List specific run types (flat list)
langsmith run list --run-type llm --limit 20 --api-key $LANGSMITH_API_KEY

</querying_traces>

All commands support these filters (all AND together):

Basic filters:

--trace-ids abc,def - Filter to specific traces
--limit N - Max results
--project NAME - Project name
--last-n-minutes N - Time filter
--since TIMESTAMP - Time filter (ISO format)
--error / --no-error - Error status
--name PATTERN - Name contains (case-insensitive)

Performance filters:

--min-latency SECONDS - Minimum latency (e.g., 5 for >= 5s)
--max-latency SECONDS - Maximum latency
--min-tokens N - Minimum total tokens
--tags tag1,tag2 - Has any of these tags

Advanced filter:

--filter QUERY - Raw LangSmith filter query for complex cases (feedback, metadata, etc.)

# Filter traces by feedback score using raw LangSmith query
langsmith trace list --filter 'and(eq(feedback_key, "correctness"), gte(feedback_score, 0.8))' --api-key $LANGSMITH_API_KEY

<export_format> Export creates .jsonl files (one run per line) with these fields:

{"run_id": "...", "trace_id": "...", "name": "...", "run_type": "...", "parent_run_id": "...", "inputs": {...}, "outputs": {...}}

Use --include-io or --full to include inputs/outputs (required for dataset generation). </export_format>

Start with traces — they provide complete context needed for trajectory and dataset generation
Use trace export --full for bulk data destined for datasets
Always specify --project to avoid mixing data from different projects
Use /tmp for temporary exports
Include --include-metadata for performance/cost analysis
Stitch files: cat ./traces/*.jsonl > all.jsonl

Account or credentials required

2 FOR YOU

All items below are required.

01
Get a LangSmith API key
Set the LANGSMITH_API_KEY environment variable (or pass --api-key on each command); optionally set LANGSMITH_PROJECT so queries default to the right project.
You set: LANGSMITH_API_KEY — or pass --api-key on each command instead
Optional; you set: LANGSMITH_PROJECT
02
Install the langsmith CLI
Install the CLI with curl -sSL https://raw.githubusercontent.com/langchain-ai/langsmith-cli/main/scripts/install.sh | sh before running trace commands.
```
curl -sSL https://raw.githubusercontent.com/langchain-ai/langsmith-cli/main/scripts/install.sh | sh
```

Listed as free: Connected services may require an account or charge fees.

This skill ships inside the LangSmith Tracing & Evaluation Skills plugin.

Install just this skill

Run this outside your AI tool, using Node.js 22.20.0 or newer. Skills CLI 1.7.2 skips skill selection; prompts cancel installation without an interactive terminal. If choosing Project when asked about installation scope, open your project's folder before running this.

Install checklist

Check selected tools if a picker appears; Claude Code may be unchecked under Additional agents. ↑/↓ moves, Space toggles selections, Enter confirms.
Without a tool picker, check Installing to: for your intended tool. Stop if it names a different tool.
If asked, choose Symlink (Recommended) for shared files or Copy to all agents for separate copies; review the summary before confirming installation.
Install the find-skills skill? defaults to Yes, adding another skill for all projects. Choose No if you don't want that extra installation.

Skills CLI 1.7.2 sends install data to skills.sh by default, including the source, skill names, target tools and file details; GitHub sources are reported only when GitHub confirms the repository is public, while other remote sources can also be reported. Confirmed-public GitHub sources also trigger a security-audit request. Before installing, set the environment variable DO_NOT_TRACK=1 or DISABLE_TELEMETRY=1 to disable both.

npx skills add langchain-ai/langsmith-skills --skill "langsmith-trace" --full-depth

After installing for Claude Code, open it in the installation project if you chose Project; Global makes the skill available across projects on this computer. Once Claude Code loads the skill, its description helps Claude decide when to use it.

If installation created a top-level skills folder that did not exist when your Claude Code session started, run /reload-skills in that session. Run it again after later changes in that folder.

Install the whole plugin

This route installs the LangSmith Tracing & Evaluation Skills plugin package selected for your AI tool.

PLUGIN MARKETPLACE

Add the marketplace, then install the plugin

Typed inside the agent's own prompt, not in a terminal. The marketplace is called langsmith-skills, which is the part after the @.

/plugin marketplace add langchain-ai/langsmith-skills

/plugin install langsmith-skills@langsmith-skills

Open plugin page ↗

What's inside

Files included with this skill.

View on GitHub

Account or credentials required

Install just this skill

Install the whole plugin

Add the marketplace, then install the plugin

What's inside

More skills

MCP keyword matches