crawl_single_pageLABSCO SUMMARY
A knowledge base you choose, not one the model inherited.
The strategies here — hybrid search, reranking, separate indexing of code examples — are the ones that decide whether RAG helps or quietly hurts, and each is a switch rather than a baked-in choice. The author describes the repository as a testbed on its way into Archon, so expect the interface to move.
What it is
An MCP server pairing the Crawl4AI crawler with a Supabase vector store, so pages you scrape become a knowledge base the assistant can search.
What you get
- Pages crawled and stored, then retrieved over — answers come from documentation you chose
- Hybrid search combining vector similarity with keyword matching
- Contextual embeddings for richer semantic matching, switchable in configuration
- Cross-encoder reranking, for when the right passage keeps coming back third
- Code examples extracted and indexed separately, so retrieval can return a snippet rather than prose about one
- A knowledge graph mode used to check generated code against a real repository, catching invented APIs
Requirements
Supabase for the vector store and OpenAI for embeddings. Each retrieval strategy is a configuration switch, off by default.
Setup effort
Build from source — git clone https://github.com/coleam00/mcp-crawl4ai-rag.git
USE CASES
01
Giving an assistant docs for a library it does not know
Crawl the page you care about into the store, then ask against it rather than against the model's memory.
02
Getting a snippet back instead of a paragraph about one
Turn on code example extraction so retrieval returns code that was actually in the docs.
03
Checking generated code against the real repository
Use the knowledge graph mode to catch calls to methods that do not exist.
04
Fixing retrieval that keeps returning near-misses
Add reranking and hybrid search rather than rewriting the prompt again.
8 tools
Listed in project docs. Your client may show a different list.
smart_crawl_urlget_available_sourcesperform_rag_querysearch_code_examplesparse_github_repositorycheck_ai_script_hallucinationsquery_knowledge_graphBEFORE YOU INSTALL
What to know first
Tools you get
8 tools named in the project's documentation.
Requires
- python on PATH
- OPENAI_API_KEY in your environment
- SUPABASE_SERVICE_KEY in your environment
- SUPABASE_URL in your environment
INSTALLATION
Connect your client
You clone and build this one yourself.
Server setup
TERMINAL
git clone https://github.com/coleam00/mcp-crawl4ai-rag.git
cd mcp-crawl4ai-rag
pip install uv
uv venv
source .venv/bin/activate
uv pip install -e .Still in the activated .venv, run Crawl4AI's setup command, which the README lists with the dependency install: crawl4ai-setup
TERMINAL
crawl4ai-setupCreate a .env file in the project root with HOST=0.0.0.0, PORT=8051, TRANSPORT=sse, OPENAI_API_KEY, MODEL_CHOICE, SUPABASE_URL and SUPABASE_SERVICE_KEY (add the USE_* strategy switches and NEO4J_URI / NEO4J_USER / NEO4J_PASSWORD only if you use them)
In your Supabase project's SQL Editor, paste the contents of crawled_pages.sql from the repository into a new query and run it
TERMINAL
uv run src/crawl4ai_mcp.pyOr, instead of uv, build and run the Docker image from the clone: docker build -t mcp/crawl4ai-rag --build-arg PORT=8051 . then docker run --env-file .env -p 8051:8051 mcp/crawl4ai-rag
MCP Clients
Claude Desktop
CLAUDE DESKTOP · JSON
{
"mcpServers": {
"crawl4ai-rag": {
"command": "python",
"args": [
"<YOUR_PATH_TO_CRAWL4AI_MCP>/src/crawl4ai_mcp.py"
],
"env": {
"TRANSPORT": "stdio",
"OPENAI_API_KEY": "<YOUR_OPENAI_API_KEY>",
"SUPABASE_URL": "<YOUR_SUPABASE_URL>",
"SUPABASE_SERVICE_KEY": "<YOUR_SUPABASE_SERVICE_KEY>",
"USE_KNOWLEDGE_GRAPH": "false",
"NEO4J_URI": "bolt://localhost:7687",
"NEO4J_USER": "neo4j",
"NEO4J_PASSWORD": "<YOUR_NEO4J_PASSWORD>"
}
}
}
}<YOUR_PATH_TO_CRAWL4AI_MCP>/src/crawl4ai_mcp.py is the absolute path to src/crawl4ai_mcp.py in your clone. The dependencies live in the clone's .venv, so if a plain python on your PATH does not have them, set command to the absolute path of .venv/bin/python inside the clone.
Windsurf
WINDSURF · JSON
{
"mcpServers": {
"crawl4ai-rag": {
"serverUrl": "http://localhost:8051/sse"
}
}
}Claude Code
CLAUDE CODE · TERMINAL
claude mcp add-json crawl4ai-rag '{"type":"sse","url":"http://localhost:8051/sse"}' --scope userAny MCP client — SSE
ANY MCP CLIENT — SSE · JSON
{
"mcpServers": {
"crawl4ai-rag": {
"type": "sse",
"url": "http://localhost:8051/sse"
}
}
}Any MCP client — Docker
ANY MCP CLIENT — DOCKER · JSON
{
"mcpServers": {
"crawl4ai-rag": {
"command": "docker",
"args": [
"run",
"--rm",
"-i",
"-e",
"TRANSPORT",
"-e",
"OPENAI_API_KEY",
"-e",
"SUPABASE_URL",
"-e",
"SUPABASE_SERVICE_KEY",
"-e",
"USE_KNOWLEDGE_GRAPH",
"-e",
"NEO4J_URI",
"-e",
"NEO4J_USER",
"-e",
"NEO4J_PASSWORD",
"mcp/crawl4ai-rag"
],
"env": {
"TRANSPORT": "stdio",
"OPENAI_API_KEY": "<YOUR_OPENAI_API_KEY>",
"SUPABASE_URL": "<YOUR_SUPABASE_URL>",
"SUPABASE_SERVICE_KEY": "<YOUR_SUPABASE_SERVICE_KEY>",
"USE_KNOWLEDGE_GRAPH": "false",
"NEO4J_URI": "bolt://localhost:7687",
"NEO4J_USER": "neo4j",
"NEO4J_PASSWORD": "<YOUR_NEO4J_PASSWORD>"
}
}
}
}Uses the mcp/crawl4ai-rag image built from the clone with the docker build step above, which exists only on your machine.
