Agent Evals by Galileo

 
 Open the Command Palette (`Ctrl + Shift + P` on Windows/Linux, or `Cmd + Shift + P` on Mac) and search for **"MCP: Open User Configuration"**
 

 
 Copy and paste the configuration below. Replace `YOUR-API-KEY` with your actual Galileo API key.

 ```json VSCode MCP Configuration highlight={3-9} theme={null}
 {
 "servers": {
 "galileo_mcp_server": {
 "url": "https://api.galileo.ai/mcp/http/mcp",
 "headers": {
 "Galileo-API-Key": "YOUR-API-KEY",
 "Accept": "text/event-stream"
 }
 }
 },
 "inputs": []
 }

 
 
 
 
 Open the Cursor command palette (`Ctrl + Shift + P` on Windows/Linux, or `Cmd + Shift + P` on Mac), then choose "Open MCP Settings".
 

{
"mcpServers": {
"galileo_mcp_server": {
"url": "https://api.galileo.ai/mcp/http/mcp",
"headers": {
"Galileo-API-Key": "YOUR-API-KEY",
"Accept": "text/event-stream"
}
}
}
}

 
 
 
 The configuration is the same for both Cursor and VS Code. Make sure to replace `YOUR-API-KEY` with your actual Galileo API key from the [API keys page](https://app.galileo.ai/settings/api-keys).

## Tools

The Galileo MCP server provides powerful tools that you can access through
natural conversation with your AI assistant. Simply ask questions or make
requests, and the AI will use these tools to help you.

 
 
 Generate synthetic datasets or upload your own data to test and evaluate your AI applications. The tool supports creating datasets with various types of queries including general queries, prompt injections, off-topic content, and toxic content scenarios.

Create a dataset with 50 customer service queries about billing issues

Generate a dataset of 30 chatbot queries, including some prompt injection attempts

Make a dataset with product recommendation queries and include off-topic questions

 
 
 Track the progress of your dataset generation and preview the generated content. You'll see the first 10 rows of data along with generation status and progress updates.

Check the status of my dataset that's currently generating

Show me the preview of dataset abc-123

Is my customer service dataset ready yet?

 
 
 Build reusable prompt templates that you can use across all your projects. Set up model configurations, temperature settings, and other parameters for consistent prompt behavior.

Create a prompt template called "Friendly Assistant" for customer support responses

Make a prompt template for summarizing technical documentation with lower temperature

Set up a chat template for a code review assistant

 
 
 Get complete guidance on setting up and running Galileo experiments, including dataset preparation, metrics configuration, and integration with your existing code. Available for Python and other supported languages.

How do I set up a Galileo experiment in Python?

Show me how to run an experiment with my RAG application

Guide me through creating an experiment for my agentic workflow

 
 
 Analyze your application's Log streams to identify issues, patterns, and opportunities for improvement. Get specific recommendations based on your logged data.

What issues do you see in my production log stream?

Fetch the most recent signals from Galileo and propose fixes for them in my code

Get insights about my chatbot log stream and suggest improvements

 
 
 Get step-by-step integration guides for adding Galileo observability to your OpenAI applications. Automatically log prompts, responses, model parameters, and token usage with minimal code changes.

How do I add Galileo logging to my OpenAI application?

Show me how to integrate Galileo with OpenAI in TypeScript

Help me set up Galileo observability for my GPT-4 chatbot

 
 
 Get complete integration instructions for adding Galileo to your LangChain applications. Capture full traces of your chains, agents, and tools with automatic logging.

How do I integrate Galileo with my LangChain application?

Show me how to add Galileo tracing to my LangChain agent

Help me log my LangChain RAG pipeline with Galileo

 
 
 Find relevant information, code examples, API references, and implementation guides across all Galileo documentation. Get direct links to the pages you need.

How do I set up data logging in Galileo?

Find documentation about custom metrics

Search for examples of agentic AI evaluation

## Example use cases

### Create a synthetic dataset

 Ask your AI assistant:

 
 ```text Query wrap theme={null}
 Create a synthetic dataset with 20 customer service queries about product
 returns. Include both general queries and off-topic queries.

How do I integrate Galileo with my LangChain application in Python?

Fetch the most recent signals from Galileo and propose fixes for them in my code

Can you show me how to add Galileo logging to my agent bot?

Help me create a synthetic dataset for customer support queries

How do I integrate Galileo with LangChain?