Create, manage, and upload evaluation datasets to LangSmith for testing and validation. Supports four dataset types: final_response (full conversations), single_step (individual node behavior), trajectory (tool call sequences), and RAG (question/chunks/answer/citations) CLI commands for dataset lifecycle management: create, list, get, delete, export, and upload from local JSON files SDK-based dataset creation in Python and JavaScript with programmatic example addition Example management...

Build evaluation pipelines for LangSmith with LLM-as-Judge and custom code evaluators. Three core components: creating evaluators (LLM-as-Judge or custom code), defining run functions to capture agent outputs and trajectories, and running evaluations locally or auto-running via uploaded evaluators Supports both offline evaluators (comparing run outputs to dataset examples) and online evaluators (real-time quality checks on production runs) Requires LangSmith API key and project...

Add tracing to LangChain/LangGraph apps and query trace data via CLI for debugging and dataset generation. Automatic tracing for LangChain/LangGraph apps with environment variables; manual tracing via @traceable decorator and wrap_openai() for other frameworks in Python and TypeScript Query traces (complete execution trees) or runs (individual nodes) using langsmith CLI with filters for time, latency, errors, tags, and custom metadata Export traces to JSONL files preserving hierarchy;...