langsmith-dataset

The set of test cases an AI system gets graded on, kept in LangSmith so that every experiment is measured against the same one.

WHEN TO USE

SCOPE

USE FOR

Turn real production traces into a reusable evaluation dataset.
Upload a local JSON file of examples to LangSmith as a new dataset.
Build a trajectory dataset that records the exact tool calls an agent should take.
Export an existing dataset to a local file so you can inspect or edit it.
Add, list, or delete individual examples inside a dataset.

This is a saved copy of the skill's SKILL.md instructions. It may differ from the upstream file or the version you install.

In this document (7)

Create, manage, and upload evaluation datasets to LangSmith for testing and validation.

Environment Variables

LANGSMITH_API_KEY=lsv2_pt_your_api_key_here          # REQUIRED
LANGSMITH_PROJECT=your-project-name                   # Check this to know which project has traces
LANGSMITH_WORKSPACE_ID=your-workspace-id              # Optional: for org-scoped keys

Authentication is REQUIRED: either set the LANGSMITH_API_KEY environment variable, or pass the --api-key flag to CLI commands (preferred):

langsmith dataset list --api-key $LANGSMITH_API_KEY

IMPORTANT: Always check the environment variables or .env file for LANGSMITH_PROJECT before querying or interacting with LangSmith. This tells you which project contains the relevant traces and data. If the LangSmith project is not available, use your best judgement to identify the right one.

Python Dependencies

pip install langsmith

JavaScript Dependencies

npm install langsmith

CLI Tool

curl -sSL https://raw.githubusercontent.com/langchain-ai/langsmith-cli/main/scripts/install.sh | sh

Use the langsmith CLI to manage datasets and examples.

Dataset Commands

langsmith dataset list - List datasets in LangSmith
langsmith dataset get <name-or-id> - View dataset details
langsmith dataset create --name <name> - Create a new empty dataset
langsmith dataset delete <name-or-id> - Delete a dataset
langsmith dataset export <name-or-id> <output-file> - Export dataset to local JSON file
langsmith dataset upload <file> --name <name> - Upload a local JSON file as a dataset

Example Commands

langsmith example list --dataset <name> - List examples in a dataset
langsmith example create --dataset <name> --inputs <json> - Add an example to a dataset
langsmith example delete <example-id> - Delete an example

Experiment Commands

langsmith experiment list --dataset <name> - List experiments for a dataset
langsmith experiment get <name> - View experiment results

Common Flags

--limit N - Limit number of results
--yes - Skip confirmation prompts (use with caution)

IMPORTANT - Safety Prompts:

The CLI prompts for confirmation before destructive operations (delete, overwrite)
If you are running with user input: ALWAYS wait for user input; NEVER use --yes unless the user explicitly requests it
If you are running non-interactively: Use --yes to skip confirmation prompts

<dataset_types_overview> Common evaluation dataset types:

final_response - Full conversation with expected output. Tests complete agent behavior.
single_step - Single node inputs/outputs. Tests specific node behavior (e.g., one LLM call or tool).
trajectory - Tool call sequence. Tests execution path (ordered list of tool names).
rag - Question/chunks/answer/citations. Tests retrieval quality. </dataset_types_overview>

<creating_datasets>

Creating Datasets

Datasets are JSON files with an array of examples. Each example has inputs and outputs.

From Exported Traces (Programmatic)

Export traces first, then process them into dataset format using code:

# 1. Export traces to JSONL files
langsmith trace export ./traces --project my-project --limit 20 --full --api-key $LANGSMITH_API_KEY

import json
from pathlib import Path
from langsmith import Client

client = Client()

# 2. Process traces into dataset examples
examples = []
for jsonl_file in Path("./traces").glob("*.jsonl"):
    runs = [json.loads(line) for line in jsonl_file.read_text().strip().split("\n")]
    root = next((r for r in runs if r.get("parent_run_id") is None), None)
    if root and root.get("inputs") and root.get("outputs"):
        examples.append({
            "trace_id": root.get("trace_id"),
            "inputs": root["inputs"],
            "outputs": root["outputs"]
        })

# 3. Save locally
with open("/tmp/dataset.json", "w") as f:
    json.dump(examples, f, indent=2)

import { Client } from "langsmith";
import { readFileSync, writeFileSync, readdirSync } from "fs";
import { join } from "path";

const client = new Client();

// 2. Process traces into dataset examples
const examples: Array<{trace_id?: string, inputs: Record<string, any>, outputs: Record<string, any>}> = [];
const files = readdirSync("./traces").filter(f => f.endsWith(".jsonl"));

for (const file of files) {
  const lines = readFileSync(join("./traces", file), "utf-8").trim().split("\n");
  const runs = lines.map(line => JSON.parse(line));
  const root = runs.find(r => r.parent_run_id == null);
  if (root?.inputs && root?.outputs) {
    examples.push({ trace_id: root.trace_id, inputs: root.inputs, outputs: root.outputs });
  }
}

// 3. Save locally
writeFileSync("/tmp/dataset.json", JSON.stringify(examples, null, 2));

Upload to LangSmith

# Upload local JSON file as a dataset
langsmith dataset upload /tmp/dataset.json --name "My Evaluation Dataset" --api-key $LANGSMITH_API_KEY

Using the SDK Directly

from langsmith import Client

client = Client()

# Create dataset and add examples in one step
dataset = client.create_dataset("My Dataset", description="Evaluation dataset")

# Pass examples as a list of dicts. The parallel `inputs=`/`outputs=` lists
# are still accepted but only as legacy keyword arguments.
client.create_examples(
    dataset_id=dataset.id,
    examples=[
        {"inputs": {"query": "What is AI?"}, "outputs": {"answer": "AI is..."}},
        {"inputs": {"query": "Explain RAG"}, "outputs": {"answer": "RAG is..."}},
    ],
)

import { Client } from "langsmith";

const client = new Client();

// Create dataset and add examples
const dataset = await client.createDataset("My Dataset", {
  description: "Evaluation dataset",
});

// Pass an array of examples. The `{ inputs, outputs, datasetName }` form is
// deprecated in the JS SDK; each example carries its own dataset_id.
await client.createExamples([
  {
    inputs: { query: "What is AI?" },
    outputs: { answer: "AI is..." },
    dataset_id: dataset.id,
  },
  {
    inputs: { query: "Explain RAG" },
    outputs: { answer: "RAG is..." },
    dataset_id: dataset.id,
  },
]);

</creating_datasets>

<dataset_structures>

Dataset Structures by Type

Final Response

{"trace_id": "...", "inputs": {"query": "What are the top genres?"}, "outputs": {"response": "The top genres are..."}}

Single Step

{"trace_id": "...", "inputs": {"messages": [...]}, "outputs": {"content": "..."}, "metadata": {"node_name": "model"}}

Trajectory

{"trace_id": "...", "inputs": {"query": "..."}, "outputs": {"expected_trajectory": ["tool_a", "tool_b", "tool_c"]}}

RAG

{"trace_id": "...", "inputs": {"question": "How do I..."}, "outputs": {"answer": "...", "retrieved_chunks": ["..."], "cited_chunks": ["..."]}}

</dataset_structures>

<script_usage>

CLI Usage

# List all datasets
langsmith dataset list --api-key $LANGSMITH_API_KEY

# Get dataset details
langsmith dataset get "My Dataset" --api-key $LANGSMITH_API_KEY

# Create an empty dataset
langsmith dataset create --name "New Dataset" --description "For evaluation" --api-key $LANGSMITH_API_KEY

# Upload a local JSON file
langsmith dataset upload /tmp/dataset.json --name "My Dataset" --api-key $LANGSMITH_API_KEY

# Export a dataset to local file
langsmith dataset export "My Dataset" /tmp/exported.json --limit 100 --api-key $LANGSMITH_API_KEY

# Delete a dataset
langsmith dataset delete "My Dataset" --api-key $LANGSMITH_API_KEY

# List examples in a dataset
langsmith example list --dataset "My Dataset" --limit 10 --api-key $LANGSMITH_API_KEY

# Add an example
langsmith example create --dataset "My Dataset" \
  --inputs '{"query": "test"}' \
  --outputs '{"answer": "result"}' --api-key $LANGSMITH_API_KEY

# List experiments
langsmith experiment list --dataset "My Dataset" --api-key $LANGSMITH_API_KEY
langsmith experiment get "eval-v1" --api-key $LANGSMITH_API_KEY

</script_usage>

<example_workflow> Complete workflow from traces to uploaded LangSmith dataset:

# 1. Export traces from LangSmith
langsmith trace export ./traces --project my-project --limit 20 --full --api-key $LANGSMITH_API_KEY

# 2. Process traces into dataset format (using Python/JS code)
# See "Creating Datasets" section above

# 3. Upload to LangSmith
langsmith dataset upload /tmp/final_response.json --name "Skills: Final Response" --api-key $LANGSMITH_API_KEY
langsmith dataset upload /tmp/trajectory.json --name "Skills: Trajectory" --api-key $LANGSMITH_API_KEY

# 4. Verify upload
langsmith dataset list --api-key $LANGSMITH_API_KEY
langsmith dataset get "Skills: Final Response" --api-key $LANGSMITH_API_KEY
langsmith example list --dataset "Skills: Final Response" --limit 3 --api-key $LANGSMITH_API_KEY

# 5. Run experiments
langsmith experiment list --dataset "Skills: Final Response" --api-key $LANGSMITH_API_KEY

</example_workflow>

Dataset upload fails:

Verify LANGSMITH_API_KEY is set
Check JSON file is valid: each element needs inputs (and optionally outputs)
Dataset name must be unique, or delete existing first with langsmith dataset delete

Empty dataset after upload:

Verify JSON file contains an array of objects with inputs key
Check file isn't empty: langsmith example list --dataset "Name"

Export has no data:

Ensure traces were exported with --full flag to include inputs/outputs
Verify traces have both inputs and outputs populated

Example count mismatch:

Use langsmith dataset get "Name" to check remote count
Compare with local file to verify upload completeness

</output>

Account or credentials required

2 FOR YOU

All items below are required.

01
Get a LangSmith API key
Set the LANGSMITH_API_KEY environment variable (or pass --api-key on each command); also set LANGSMITH_PROJECT so the CLI knows which project holds the traces you want to export.
You set: LANGSMITH_API_KEY — or pass --api-key on each command instead
Optional; you set: LANGSMITH_PROJECT
02
Install the langsmith CLI
Install the CLI with curl -sSL https://raw.githubusercontent.com/langchain-ai/langsmith-cli/main/scripts/install.sh | sh before running any dataset commands.
```
curl -sSL https://raw.githubusercontent.com/langchain-ai/langsmith-cli/main/scripts/install.sh | sh
```

Listed as free: Connected services may require an account or charge fees.

This skill ships inside the LangSmith Tracing & Evaluation Skills plugin.

Install just this skill

Run this outside your AI tool, using Node.js 22.20.0 or newer. Skills CLI 1.7.2 skips skill selection; prompts cancel installation without an interactive terminal. If choosing Project when asked about installation scope, open your project's folder before running this.

Install checklist

Check selected tools if a picker appears; Claude Code may be unchecked under Additional agents. ↑/↓ moves, Space toggles selections, Enter confirms.
Without a tool picker, check Installing to: for your intended tool. Stop if it names a different tool.
If asked, choose Symlink (Recommended) for shared files or Copy to all agents for separate copies; review the summary before confirming installation.
Install the find-skills skill? defaults to Yes, adding another skill for all projects. Choose No if you don't want that extra installation.

Skills CLI 1.7.2 sends install data to skills.sh by default, including the source, skill names, target tools and file details; GitHub sources are reported only when GitHub confirms the repository is public, while other remote sources can also be reported. Confirmed-public GitHub sources also trigger a security-audit request. Before installing, set the environment variable DO_NOT_TRACK=1 or DISABLE_TELEMETRY=1 to disable both.

npx skills add langchain-ai/langsmith-skills --skill "langsmith-dataset" --full-depth

After installing for Claude Code, open it in the installation project if you chose Project; Global makes the skill available across projects on this computer. Once Claude Code loads the skill, its description helps Claude decide when to use it.

If installation created a top-level skills folder that did not exist when your Claude Code session started, run /reload-skills in that session. Run it again after later changes in that folder.

Install the whole plugin

This route installs the LangSmith Tracing & Evaluation Skills plugin package selected for your AI tool.

PLUGIN MARKETPLACE

Add the marketplace, then install the plugin

Typed inside the agent's own prompt, not in a terminal. The marketplace is called langsmith-skills, which is the part after the @.

/plugin marketplace add langchain-ai/langsmith-skills

/plugin install langsmith-skills@langsmith-skills

Open plugin page ↗

What's inside

Files included with this skill.

View on GitHub