langsmith-evaluator

CRITICAL: Before writing ANY evaluator or extraction logic, you MUST:

1. Run your agent on sample inputs and capture the actual output
2. Inspect the output - print it, query LangSmith traces, understand the exact structure
3. Only then write code that processes that output

Output structures vary significantly by framework, agent type, and configuration. Never assume the shape - always verify first. Query LangSmith traces to when outputs don't contain needed data to understand how to extract from execution. </crucial_requirement>

<evaluator_format>

Offline Evaluators (attached to datasets):

• Function signature: (run, example) - receives both run outputs and dataset example
• Use case: Comparing agent outputs to expected values in a dataset
• Upload with: --dataset "Dataset Name"

Online Evaluators (attached to projects):

• Function signature: (run) - receives only run outputs, NO example parameter
• Use case: Real-time quality checks on production runs (no reference data)
• Upload with: --project "Project Name"

CRITICAL - Return Format:

• Each evaluator returns ONE metric only. For multiple metrics, create multiple evaluator functions.
• Do NOT return {"metric_name": value} or lists of metrics - this will error.

CRITICAL - Local vs Uploaded Differences:

<evaluator_types>

• LLM as Judge - Uses an LLM to grade outputs. Best for subjective quality (accuracy, helpfulness, relevance).
• Custom Code - Deterministic logic. Best for objective checks (exact match, trajectory validation, format compliance). </evaluator_types>

<llm_judge>

Use langsmith evaluator create-llm to create a server-managed LLM-as-judge run rule. It requires --model-config plus exactly one target (--dataset or --project). Supply either --prompt and --schema JSON files or a Prompt Hub reference with --hub-ref.

Before writing an evaluator:

1. Inspect your dataset to understand expected field names (see Golden Rule above)
2. Test your run function and verify its output structure matches the dataset schema
3. Query LangSmith traces to debug any mismatches

def trajectory_evaluator(run, example):
    run_outputs = run.outputs if hasattr(run, "outputs") else run.get("outputs", {}) or {}
    example_outputs = example.outputs if hasattr(example, "outputs") else example.get("outputs", {}) or {}
    # IMPORTANT: Replace these placeholders with your actual field names
    # 1. Query your LangSmith trace to see what fields exist in run outputs
    # 2. Check your dataset schema for expected field names
    # Note: Trajectory data may not appear in default output - verify against trace!
    actual = run_outputs.get("YOUR_TRAJECTORY_FIELD", [])
    expected = example_outputs.get("YOUR_EXPECTED_FIELD", [])
    return {"score": 1 if actual == expected else 0, "comment": f"Expected {expected}, got {actual}"}

function trajectoryEvaluator(run, example) {
    const runOutputs = run.outputs ?? {};
    const exampleOutputs = example.outputs ?? {};
    // IMPORTANT: Replace these placeholders with your actual field names
    // 1. Query your LangSmith trace to see what fields exist in run outputs
    // 2. Check your dataset schema for expected field names
    const actual = runOutputs.YOUR_TRAJECTORY_FIELD ?? [];
    const expected = exampleOutputs.YOUR_EXPECTED_FIELD ?? [];
    const match = JSON.stringify(actual) === JSON.stringify(expected);
    return { score: match ? 1 : 0, comment: `Expected ${JSON.stringify(expected)}, got ${JSON.stringify(actual)}` };
}

</code_evaluators>

<run_functions>

Run functions execute your agent and return outputs for evaluation.

CRITICAL - Test Your Run Function First: Before writing evaluators, you MUST test your run function and inspect the actual output structure. Output shapes vary by framework, agent type, and configuration.

Debugging workflow:

1. Run your agent once on sample input
2. Query the trace to see the execution structure
3. Print the raw output and verify against trace to output contains the right data
4. Adjust the run function as needed
5. Verify your output matches your dataset schema

Try your hardest to match your run function output to your dataset schema. This makes evaluators simple and reusable. If matching isn't possible, your evaluator must know how to extract and compare the right fields from each side.

def run_agent(inputs: dict) -> dict:
    result = your_agent.run(inputs)
    # ALWAYS inspect output shape first - run this, check the print, query traces
    print(f"DEBUG - type: {type(result)}, keys: {result.keys() if hasattr(result, 'keys') else 'N/A'}")
    print(f"DEBUG - value: {result}")
    return {"output": result}  # Adjust to match your dataset schema

async function runAgent(inputs) {
    const result = await yourAgent.invoke(inputs);
    // ALWAYS inspect output shape first
    console.log("DEBUG - type:", typeof result, "keys:", Object.keys(result));
    console.log("DEBUG - value:", result);
    return { output: result };  // Adjust to match your dataset schema
}

IMPORTANT - Auto-Run Behavior: Evaluators uploaded to a dataset automatically run when you run experiments on that dataset. You do NOT need to pass them to evaluate() - just run your agent against the dataset and the uploaded evaluators execute automatically.

IMPORTANT - Local vs Uploaded: Uploaded evaluators run in a sandboxed environment with very limited package access. Only use built-in/standard library imports, and place all imports inside the evaluator function body. For dataset (offline) evaluators, prefer running locally with evaluate(evaluators=[...]) first — this gives you full package access.

IMPORTANT - Code vs Structured Evaluators:

• Code evaluators: Upload with langsmith evaluator upload. They run in a limited environment without external packages and work well for deterministic logic.
• Structured evaluators (LLM-as-Judge): Create with langsmith evaluator create-llm using a model config and either prompt/schema files or --hub-ref.

IMPORTANT - Choose the right target:

• --dataset: Offline evaluator with (run, example) signature - for comparing to expected values
• --project: Online evaluator with (run) signature - for real-time quality checks

You must specify one. Global evaluators are not supported.

# List all attached evaluator rules
langsmith evaluator list

# Inspect matching rules by display name
langsmith evaluator get "Trajectory Match"

# Inspect project rules by session ID (not project name)
langsmith evaluator get --session-id <project-session-id>

# Upload offline evaluator (attached to dataset)
langsmith evaluator upload \
  --name "Trajectory Match" \
  --function trajectory_evaluator \
  --dataset "My Dataset" \
  my_evaluators.py

# Upload online evaluator (attached to project)
langsmith evaluator upload \
  --name "Quality Check" \
  --function quality_check \
  --project "Production Agent" \
  my_evaluators.py

# Replace an existing rule with the same name and target (prompts first)
langsmith evaluator upload \
  --name "Trajectory Match" \
  --function trajectory_evaluator \
  --dataset "My Dataset" \
  --replace \
  my_evaluators.py

# Delete by display name (prompts first)
langsmith evaluator delete "Trajectory Match"

IMPORTANT - Safety Prompts:

• upload --replace and create-llm --replace patch the matching rule and prompt first
• delete NAME deletes every rule in the workspace with that display name, potentially across multiple targets; run get NAME first and inspect all matches
• NEVER use --yes flag unless the user explicitly requests it

Uploaded evaluators auto-run when you run experiments - no code needed. Local evaluators are passed directly for development/testing.

from langsmith import evaluate

# Uploaded evaluators run automatically
results = evaluate(run_agent, data="My Dataset", experiment_prefix="eval-v1")

# Or pass local evaluators for testing
results = evaluate(run_agent, data="My Dataset", evaluators=[my_evaluator], experiment_prefix="eval-v1")

import { evaluate } from "langsmith/evaluation";

// Uploaded evaluators run automatically
const results = await evaluate(runAgent, {
  data: "My Dataset",
  experimentPrefix: "eval-v1",
});

// Or pass local evaluators for testing
const results = await evaluate(runAgent, {
  data: "My Dataset",
  evaluators: [myEvaluator],
  experimentPrefix: "eval-v1",
});

</running_evaluations>

Output doesn't match what you expect: Query the LangSmith trace. It shows exact inputs/outputs at each step - compare what you find to what you're trying to extract.

One metric per evaluator: Return {"score": value, "comment": "..."}. For multiple metrics, create separate functions.

Field name mismatch: Your run function output must match dataset schema exactly. Inspect dataset first with client.read_example(example_id).

RunTree vs dict (Python only): Local evaluate() passes RunTree, uploaded evaluators receive dict. Handle both:

run_outputs = run.outputs if hasattr(run, "outputs") else run.get("outputs", {}) or {}

TypeScript always uses attribute access: run.outputs?.field

• LangSmith Evaluation Concepts
• Custom Code Evaluators
• OpenEvals - Readymade Evaluators