
langchain-ai / skills-benchmarks
★ 108A skill package that teaches your agent 15 capabilities — every one documented and browsable below, no GitHub required · by langchain-ai.
Each skill below is one capability this package teaches your agent. Install the whole package, or open a skill to install just that one.
INVOKE THIS SKILL when building ANY Deep Agents application. Covers create_deep_agent(), harness architecture, SKILL.md format, and configuration options.
1 file — installable on its own
INVOKE THIS SKILL when your Deep Agent needs memory, persistence, or filesystem access. Covers StateBackend (ephemeral), StoreBackend (persistent),…
1 file — installable on its own
INVOKE THIS SKILL when using subagents, task planning, or human approval in Deep Agents. Covers SubAgentMiddleware, TodoList for planning, and HITL interrupts.
1 file — installable on its own
INVOKE THIS SKILL at the START of any LangChain/LangGraph/Deep Agents project, before writing any agent code. Determines which framework layer is right for the…
1 file — installable on its own
INVOKE THIS SKILL when setting up a new project or when asked about package versions, installation, or dependency management for LangChain, LangGraph,…
1 file — installable on its own
Create LangChain agents with create_agent, define tools, and use middleware for human-in-the-loop and error handling.
1 file — installable on its own
INVOKE THIS SKILL when you need human-in-the-loop approval, custom middleware, or structured output. Covers HumanInTheLoopMiddleware for human approval of…
1 file — installable on its own
ALWAYS START HERE for any LangChain, Deep Agents, or LangGraph agent building project. Required starting point before choosing other skills or writing any…
1 file — installable on its own
langchain-rag — an installable skill for AI agents, published by langchain-ai/skills-benchmarks.
1 file — installable on its own
INVOKE THIS SKILL when writing ANY LangGraph code. Covers StateGraph, state schemas, nodes, edges, Command, Send, invoke, streaming, and error handling.
1 file — installable on its own
INVOKE THIS SKILL when implementing human-in-the-loop patterns, pausing for approval, or handling errors in LangGraph. Covers interrupt(), Command(resume=...),…
1 file — installable on its own
INVOKE THIS SKILL when your LangGraph needs to persist state, remember conversations, travel through history, or configure subgraph checkpointer scoping.…
1 file — installable on its own
INVOKE THIS SKILL when creating evaluation datasets, uploading datasets to LangSmith, or managing existing datasets. Covers dataset types (final_response,…
1 file — installable on its own
INVOKE THIS SKILL when building evaluation pipelines for LangSmith. Covers three core components: (1) Creating Evaluators - LLM-as-Judge, custom code; (2)…
1 file — installable on its own
INVOKE THIS SKILL when working with LangSmith tracing OR querying traces. Covers adding tracing to applications and querying/exporting trace data. Uses the…
24 files — installable on its own
Claude Code Skill Benchmarks
Measures how skill documentation design affects Claude Code's adherence to recommended patterns.
Note: Tests default to Claude Sonnet 4.6. Override with
BENCH_CC_MODELenv var (e.g.,BENCH_CC_MODEL=claude-opus-4-5-20251101).
Quick Start
# Setup
uv sync # Python
npm install # TypeScript
# Run a specific task with a treatment
uv run pytest tests/tasks/test_tasks.py --task=lc-basic --treatment=ALL_MAIN_SKILLS -v
# Run task with its default treatments
uv run pytest tests/tasks/test_tasks.py --task=ls-multiskill-basic -v
# Run with repetitions and parallel workers
uv run pytest tests/tasks/test_tasks.py --task=lc-basic --treatment=CONTROL,ALL_MAIN_SKILLS --count=3 -n 4 -vRequirements
- Python 3.11+ / Node.js 20+
- Docker (for sandboxed execution)
- Claude Code CLI (
claude) - API keys:
OPENAI_API_KEY,LANGSMITH_API_KEY,ANTHROPIC_API_KEY - macOS:
brew install coreutils(providesgtimeout)
Project Structure
tasks/ # Self-contained benchmark tasks
ls-lang-tracing/ # Each task has its own directory
instruction.md # Task prompt with {variable} placeholders
task.toml # Metadata + validation config
environment/ # Docker context (Dockerfile, source code)
validation/ # Test scripts (run inside Docker)
data/ # Ground truth, test cases (optional)
treatments/ # Centralized treatment definitions
common/ # Shared treatments (CONTROL, ALL_MAIN_SKILLS)
langsmith/ # LangSmith-specific treatments (LS_*)
langchain_concise/ # LangChain concise treatments (LCC_*)
oss_split/ # OSS split skill treatments (OSSS_*)
oss_merged/ # OSS merged skill treatments (OSSM_*)
skills/
main/ # Production skills
benchmarks/ # Benchmark skill variations
noise/ # Distractor skills for interference tests
scaffold/
python/ # Python scaffold
validation/runner.py # TestRunner for writing check functions
validation/core.py # Validation helpers (patterns, skills, noise)
utils.py # Docker orchestration
tasks.py # Task loader (reads task.toml)
typescript/ # TypeScript scaffold (mirrors Python)
validation/runner.ts # TestRunner (same API as Python)
validation/core.ts # Validation helpers
utils.ts # Docker orchestration
tasks.ts # Task loader
tests/
tasks/
test_tasks.py # Main test runner (pytest)
test_tasks.test.ts # Main test runner (vitest)
scripts/ # Script unit tests (Python/TypeScript parity)Tasks
Tasks are decoupled from treatments — any treatment can be used with any task. Each task defines default_treatments in its task.toml.
| Task | Category | Description |
|---|---|---|
lc-basic | langchain | SQL analytics agent |
lc-deps-tavily | langchain | Fix broken LangChain dependencies |
lc-framework-qa-agent | langchain | Framework selection: basic react agent (create_agent) |
lc-framework-approval-pipeline | langchain | Framework selection: deterministic routing (StateGraph) |
lc-framework-middleware-agent | langchain | Framework selection: pre/post hooks (create_agent) |
lc-framework-research-assistant | langchain | Framework selection: planner + sub-agents (create_deep_agent) |
lc-framework-personal-assistant | langchain | Framework selection: planning + memory (create_deep_agent) |
lc-framework-hybrid-pipeline | langchain | Framework selection: Deep Agents + LangGraph (CompiledSubAgent) |
lc-ecosystem-env-setup | langchain | Ecosystem setup: .env.example with Anthropic + LangSmith observability vars |
lc-ecosystem-install | langchain | Ecosystem setup: pick Deep Agents, write agent.py + install.sh (uv add) |
lc-ecosystem-bootstrap | langchain | Ecosystem bootstrap: pick LangGraph, write install.sh + .env.example |
ls-lang-tracing | langsmith | Add LangSmith tracing to Python/TypeScript agents |
ls-lang-evaluator | langsmith | Create LangSmith evaluators from datasets |
ls-multiskill-basic | langsmith | Create trajectory dataset from traces |
ls-multiskill-advanced | langsmith | Create dataset + evaluator pipeline |
oss-fix-lg-persistence | langgraph | Fix LangGraph persistence bugs |
oss-fix-lg-execution | langgraph | Fix LangGraph parallel execution bugs |
oss-fix-lc-streaming | langchain | Fix LangChain streaming bugs |
oss-fix-lc-hitl | langchain | Fix LangChain human-in-the-loop bugs |
oss-fix-da-memory | deepagents | Fix Deep Agents memory bugs |
Treatments
Treatments define skill configurations. They're organized by category:
| Prefix | Category | Description |
|---|---|---|
CONTROL | common | No skills (baseline) |
ALL_MAIN_SKILLS | common | All production skills |
LS_* | langsmith | LangSmith skill variations |
LCC_* | langchain_concise | LangChain CLAUDE.md and guidance tests |
OSSS_* | oss_split | OSS split skill combinations |
OSSM_* | oss_merged | OSS merged skill combinations |
Running Tests
Python (pytest) — recommended for benchmark runs
# Run specific task + treatment(s)
uv run pytest tests/tasks/test_tasks.py --task=lc-basic --treatment=ALL_MAIN_SKILLS -v
# Multiple treatments (comma-separated)
uv run pytest tests/tasks/test_tasks.py --task=lc-basic --treatment=CONTROL,ALL_MAIN_SKILLS -v
# Wildcard patterns
uv run pytest tests/tasks/test_tasks.py --task=lc-basic --treatment=LCC_* -v
# Run task with default treatments
uv run pytest tests/tasks/test_tasks.py --task=ls-multiskill-basic -v
# With repetitions and parallel workers
uv run pytest tests/tasks/test_tasks.py --task=lc-basic --treatment=CONTROL --count=3 -n 4 -v
# List all available combinations
uv run pytest tests/tasks/test_tasks.py --collect-onlyTypeScript (vitest)
The vitest runner executes the same validation pipeline and is useful for setup verification and TypeScript development. Use pytest for benchmark runs — vitest threads cannot parallelize Docker execution, so multiple treatments run sequentially.
# Run specific task + treatment
TASK=lc-basic TREATMENT=ALL_MAIN_SKILLS npx vitest run tests/tasks/test_tasks.test.ts
# List test cases without running (like pytest --collect-only)
npx vitest list tests/tasks/test_tasks.test.tsHow It Works
- Setup: Creates isolated temp directory with skills, CLAUDE.md, and environment files
- Run: Executes Claude Code CLI in Docker (
--dangerously-skip-permissionsfor headless operation) - Validate: Runs test scripts in Docker against Claude's artifacts (config-driven via
task.toml) - Report: Logs results to local experiment directory and LangSmith
- Cleanup: Removes temp directory and test resources (LangSmith datasets/projects)
Results
LangSmith
Results are tracked as LangSmith experiments. Each pytest invocation creates an experiment under the skills-benchmark dataset, where every task/treatment combination becomes a row with logged inputs, outputs, and feedback scores (pass rate, duration, turns). When TRACE_TO_LANGSMITH=true, Claude Code traces are nested under experiment rows — click a row to see the full session trace (all turns, LLM calls, and tool calls). See .env.example for configuration.
Local
Results are also saved to logs/experiments/<experiment_id>/:
logs/experiments/experiment_20260217_143052/
summary.md # Results table with pass rates
metadata.json # Experiment config
events/ # Parsed events per run
raw/ # Raw Claude CLI output
reports/ # Validation reports
artifacts/ # Files Claude created (not infrastructure)Example summary output:
Treatment Checks Turns Dur Skills
----------------------------------------------------------------------------
ALL_MAIN_SKILLS 9/9 (100%) 11 45s langchain-fundamentals
CONTROL 8/11 (73%) 14 78s noneLangSmith Tasks
Tasks prefixed with ls- query and create LangSmith resources. Important considerations:
Note: Parallel execution
Parallel execution via pytest-xdist (
-n 4) is tested and safe — each worker gets isolated LangSmith projects. Running multiple separate pytest processes simultaneously is untested and may have issues.
Warning: Orphaned resources on interrupt
If tests are interrupted (Ctrl+C), LangSmith resources may not be cleaned up:
- Projects: Delete
benchmark-*projects older than a few hours- Datasets: Delete
bench-*datasets with UUID suffixesNormal test completion auto-cleans these resources.
Contributing
See CONTRIBUTING.md for adding new tasks, skills, treatments, and test scripts.
Install the whole package (15 skills):
npx skills add https://github.com/langchain-ai/skills-benchmarksOr install a single skill:
npx skills add https://github.com/langchain-ai/skills-benchmarks --skill <name>Pick the skill name from the Skills tab — each entry there installs independently.