Labsco
MCP SERVER

Crawl4AI RAG

by coleam00

Crawl the documentation you need into a vector store, and let the assistant retrieve from it instead of guessing.

Vector Stores & RAG Retrieval
Summary
A knowledge base you choose, not one the model inherited.

The strategies here — hybrid search, reranking, separate indexing of code examples — are the ones that decide whether RAG helps or quietly hurts, and each is a switch rather than a baked-in choice. The author describes the repository as a testbed on its way into Archon, so expect the interface to move.

What it is

An MCP server pairing the Crawl4AI crawler with a Supabase vector store, so pages you scrape become a knowledge base the assistant can search.

What you get
  • Pages crawled and stored, then retrieved over — answers come from documentation you chose
  • Hybrid search combining vector similarity with keyword matching
  • Contextual embeddings for richer semantic matching, switchable in configuration
  • Cross-encoder reranking, for when the right passage keeps coming back third
  • Code examples extracted and indexed separately, so retrieval can return a snippet rather than prose about one
  • A knowledge graph mode used to check generated code against a real repository, catching invented APIs
Requirements

Supabase for the vector store and OpenAI for embeddings. Each retrieval strategy is a configuration switch, off by default.

Setup effort

Build from source — clone the repository and build it, then point your client at the binary