LangSmith MCP Server

Call this tool when you need to understand how to create datasets in LangSmith.

Fetch LangSmith runs from one or more projects with flexible filters and automatic pagination.

All results are paginated by character budget to keep responses manageable. Use page_number and total_pages from the response to iterate through multiple pages.

--- 📤 RETURNS (always paginated) ------------------------------ Dict with: runs, page_number, total_pages, max_chars_per_page, preview_chars. May include _truncated, _truncated_message, _truncated_preview if content exceeded budget.

--- ⚙️ PARAMETERS (required) ------------------------ project_name : str The project name to fetch runs from. For multiple projects, use JSON array string: '["project1", "project2"]'

limit : int (required) Maximum number of runs to fetch from LangSmith API (capped at 100). These runs are then paginated by character budget into pages.

page_number : int, default 1 1-based page index. Use with total_pages from response to iterate through pages.

trace_id : str, optional Return only runs that belong to this trace UUID. Example: "123e4567-e89b-12d3-a456-426614174000"

run_type : str, optional Filter runs by type: "llm", "chain", "tool", "retriever".

error : str, optional "true" for errored runs, "false" for successful runs.

is_root : str, optional "true" for only top-level traces, "false" to exclude roots.

filter : str, optional Filter Query Language (FQL) expression. Common fields: id, name, run_type, start_time, end_time, latency, total_tokens, error, tags, feedback_key, feedback_score, metadata_key, metadata_value, execution_order.

Operators: eq, neq, gt, gte, lt, lte, has, search, and, or, not.

Examples: 'gt(latency, "5s")' # runs > 5 seconds 'neq(error, null)' # errored runs 'has(tags, "beta")' # tagged "beta" 'and(eq(name,"ChatOpenAI"), eq(run_type,"llm"))' # name AND type 'search("image classification")' # full-text search

trace_filter : str, optional Filter on the root run of each trace tree. Example: 'and(eq(feedback_key,"user_score"), eq(feedback_score,1))'

tree_filter : str, optional Filter on any run in the trace tree (siblings/children included). Example: 'eq(name,"ExpandQuery")'

order_by : str, default "-start_time" Sort field; prefix "-" for descending. Examples: "-start_time", "latency"

reference_example_id : str, optional Filter runs by dataset example ID.

max_chars_per_page : int, default 25000 Max JSON character count per page (capped at 30000). Pagination splits by this budget.

preview_chars : int, default 150 Truncate long strings to this length with "… (+N chars)". Keeps responses readable.

--- 🧪 EXAMPLES ----------- 1️⃣ Get latest 10 root runs: fetch_runs("my-project", limit=10, page_number=1, is_root="true")

2️⃣ Get errored tool runs: fetch_runs("my-project", limit=50, page_number=1, run_type="tool", error="true")

3️⃣ Runs > 5s with "experimental" tag: fetch_runs("my-project", limit=50, page_number=1, filter='and(gt(latency,"5s"), has(tags,"experimental"))')

4️⃣ All runs for a trace (paginate through pages): r = fetch_runs("my-project", limit=50, page_number=1, trace_id="abc-123") # Check r["total_pages"] and fetch subsequent pages if needed

Fetch organization billing usage (trace counts) with workspace names inline.

Returns metrics from GET /api/v1/orgs/current/billing/usage. Each metric's groups is augmented to { workspace_uuid: { "workspace_name": "<name>", "value": <number> } }. If workspace is provided (UUID or display name), only that workspace's entries are included in each metric's groups.

Get a specific prompt by its exact name.

Retrieve one page of message history for a specific conversation thread.

Uses char-based pagination: pages are built by character budget (max_chars_per_page). Long strings are truncated to preview_chars. Supply page_number (1-based) on every call; use the returned total_pages to request further pages.

Fetch LangSmith datasets.

Note: If no arguments are provided, all datasets will be returned.

Fetch examples from a LangSmith dataset with advanced filtering options.

Note: Either dataset_id, dataset_name, or example_ids must be provided. If multiple are provided, they are used in order of precedence: example_ids, dataset_id, dataset_name.

List LangSmith experiment projects (reference projects) with mandatory dataset filtering.

Fetches experiment projects from LangSmith that are associated with a specific dataset. These are projects used for model evaluation and comparison. Requires either a dataset ID or dataset name to filter experiments.

--- 🧩 PURPOSE ---------- This function provides a convenient way to list and explore LangSmith experiment projects. It supports:

• Filtering experiments by reference dataset (mandatory)
• Filtering projects by name (partial match)
• Limiting the number of results
• Automatically extracting deployment IDs from nested project data
• Returns simplified project information with key metrics (latency, cost, feedback stats)

--- ⚙️ PARAMETERS ------------- reference_dataset_id : str, optional The ID of the reference dataset to filter experiments by. Either this OR reference_dataset_name must be provided (but not both).

reference_dataset_name : str, optional The name of the reference dataset to filter experiments by. Either this OR reference_dataset_id must be provided (but not both).

limit : int, default 5 Maximum number of experiments to return. This can be adjusted by agents or users based on their needs.

project_name : str, optional Filter projects by name using partial matching. If provided, only projects whose names contain this string will be returned. Example: project_name="Chat" will match "Chat-LangChain", "ChatBot", etc.

--- 📤 RETURNS ---------- Dict[str, Any] A dictionary containing an "experiments" key with a list of simplified experiment project dictionaries:

{
    "experiments": [
        {
            "name": "Experiment-Chat-LangChain",
            "experiment_id": "787d5165-f110-43ff-a3fb-66ea1a70c971",
            "feedback_stats": {...},  # Feedback statistics if available
            "latency_p50_seconds": 1.626,  # 50th percentile latency in seconds
            "latency_p99_seconds": 2.390,   # 99th percentile latency in seconds
            "total_cost": 0.00013005,       # Total cost in dollars
            "prompt_cost": 0.00002085,      # Prompt cost in dollars
            "completion_cost": 0.0001092,   # Completion cost in dollars
            "agent_deployment_id": "deployment-123"  # Only if available
        },
        ...
    ]
}

--- 🧪 EXAMPLES ------------ 1️⃣ List experiments for a dataset by ID

experiments = list_experiments(reference_dataset_id="f5ca13c6-96ad-48ba-a432-ebb6bf94528f")

2️⃣ List experiments for a dataset by name

experiments = list_experiments(reference_dataset_name="my-dataset", limit=10)

3️⃣ Find experiments with specific name pattern

experiments = list_experiments(
    reference_dataset_id="f5ca13c6-96ad-48ba-a432-ebb6bf94528f",
    project_name="Chat",
    limit=1
)

--- 🧠 NOTES FOR AGENTS --------------------

• Returns simplified experiment information with key metrics (latency, cost, feedback stats)
• The agent_deployment_id field is automatically extracted from nested project data when available, making it easy to identify agent deployments
• Experiments are filtered to include only reference projects (associated with datasets)
• The function uses name_contains for filtering, so partial matches work
• You must provide either reference_dataset_id OR reference_dataset_name, but not both
• Experiment projects are used for model evaluation and comparison across different runs

List LangSmith projects with optional filtering and detail level control.

Fetches projects from LangSmith, optionally filtering by name and controlling the level of detail returned. Can return either simplified project information or full project details. In case a dataset id or name is provided, you don't need to provide a project name.

--- 🧩 PURPOSE ---------- This function provides a convenient way to list and explore LangSmith projects. It supports:

• Filtering projects by name (partial match)
• Limiting the number of results
• Choosing between simplified or full project information
• Automatically extracting deployment IDs from nested project data

--- ⚙️ PARAMETERS ------------- limit : int, default 5 Maximum number of projects to return (as string, e.g., "5"). This can be adjusted by agents or users based on their needs.

project_name : str, optional Filter projects by name using partial matching. If provided, only projects whose names contain this string will be returned. Example: project_name="Chat" will match "Chat-LangChain", "ChatBot", etc.

more_info : str, default "false" Controls the level of detail returned:

• "false" (default): Returns simplified project information with only essential fields: name, project_id, and agent_deployment_id (if available)
• "true": Returns full project details as returned by the LangSmith API

reference_dataset_id : str, optional The ID of the reference dataset to filter projects by. Either this OR reference_dataset_name must be provided (but not both).

reference_dataset_name : str, optional The name of the reference dataset to filter projects by. Either this OR reference_dataset_id must be provided (but not both).

--- 📤 RETURNS ---------- List[dict] A list of project dictionaries. The structure depends on more_info:

When more_info=False (simplified):

[
    {
        "name": "Chat-LangChain",
        "project_id": "787d5165-f110-43ff-a3fb-66ea1a70c971",
        "agent_deployment_id": "deployment-123"  # Only if available
    },
    ...
]

When more_info=True (full details): Returns complete project objects with all fields from the LangSmith API, including metadata, settings, statistics, and nested structures.

--- 🧪 EXAMPLES ------------ 1️⃣ List first 5 projects (simplified)

projects = list_projects(limit="5")

2️⃣ Search for projects with "Chat" in the name

projects = list_projects(project_name="Chat", limit="10")

3️⃣ Get full project details

projects = list_projects(limit="3", more_info="true")

4️⃣ Find a specific project with full details

projects = list_projects(project_name="MyProject", more_info="true", limit="1")

--- 🧠 NOTES FOR AGENTS --------------------

• Use more_info="false" for quick project discovery and listing
• Use more_info="true" when you need detailed project information
• The agent_deployment_id field is automatically extracted from nested

project data when available, making it easy to identify agent deployments

• Projects are filtered to exclude reference projects by default
• The function uses name_contains for filtering, so partial matches work

Fetch prompts from LangSmith with optional filtering.

Call this tool when you need to understand how to create and push prompts to LangSmith.

Read a specific dataset from LangSmith.

Note: Either dataset_id or dataset_name must be provided to identify the dataset. If both are provided, dataset_id takes precedence.

Read a specific example from LangSmith.

Call this tool when you need to understand how to run experiments and evaluations in LangSmith.

Call this tool when you need to understand how to update dataset examples in LangSmith.