Labsco
yoanbernabeu logo

S3 Documentation MCP Server

★ 7

from yoanbernabeu

A lightweight Model Context Protocol (MCP) server that brings RAG (Retrieval-Augmented Generation) capabilities to your LLM over Markdown documentation stored on S3.

🔥🔥🔥✓ VerifiedAccount requiredAdvanced setup
#automation#database#documents#cloud#devops#git#security#monitoring

S3 Documentation MCP Server

CI codecov Build and Push Docker Image Docker Hub

A lightweight Model Context Protocol (MCP) server that brings RAG (Retrieval-Augmented Generation) capabilities to your LLM over Markdown documentation stored on S3.

Built for simplicity:

  • 🪶 Lightweight Stack: No heavy dependencies or cloud services
  • 🏠 Flexible Embeddings: Choose between Ollama (local, free) or OpenAI (cloud, high-accuracy)
  • 💾 File-based Storage: Vector indices stored as simple files (HNSWLib)
  • 🔌 S3-Compatible: Works with any S3-compatible storage (AWS, MinIO, Scaleway, Cloudflare R2...)

[!IMPORTANT]
🚧 This project is a work in progress. APIs and behavior may change at any time, and backward compatibility is not ensured. Not suitable for production.

Use Cases

  • 📚 Product Documentation: Let Claude/Cursor/etc answer from your docs
  • 🏢 Internal Wiki: AI-powered company knowledge search
  • 📖 API Docs: Help developers find API information
  • 🎓 Educational Content: Build AI tutors with course materials

Connect to MCP Clients

Once your server is running, you need to configure your MCP client to connect to it.

Cursor

Edit your ~/.cursor/mcp.json file and add:

{
  "mcpServers": {
    "doc": {
        "type": "streamable-http",
        "url": "http://127.0.0.1:3000/mcp",
        "note": "S3 Documentation RAG Server"
    }
  }
}

Claude Desktop

Edit your Claude Desktop configuration file:

  • macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
  • Windows: %APPDATA%/Claude/claude_desktop_config.json
{
  "mcpServers": {
    "doc": {
        "type": "streamable-http",
        "url": "http://127.0.0.1:3000/mcp",
        "note": "S3 Documentation RAG Server"
    }
  }
}

Restart your MCP client, and you should now see:

  • 3 MCP Tools: search_documentation, refresh_index, get_full_document
  • MCP Resources: Full list of indexed documentation files with direct access

💡 Tip: If using Docker, make sure the port mapping matches your configuration (default is 3000:3000)

Features

  • 🔌 Universal S3: AWS S3, MinIO, Scaleway, DigitalOcean Spaces, Cloudflare R2, Wasabi...
  • 🧠 Flexible Embeddings:
    • Ollama (nomic-embed-text) - Local, free, offline-capable
    • OpenAI (text-embedding-3-small, text-embedding-3-large) - Cloud-based, high-accuracy, multilingual
  • 🔄 Smart Sync: Incremental updates via ETag comparison + automatic full sync on empty vector store
  • ⚡ Fast Search: HNSWLib vector index with cosine similarity
  • 🔐 Optional Auth: API key authentication for secure deployments
  • 🛠️ 3 MCP Tools: search_documentation, refresh_index, and get_full_document
  • 📚 MCP Resources: Native support for discovering and reading indexed files via standard MCP Resources API

How It Works

The server follows a simple pipeline:

  1. S3Loader: Scans your S3 bucket for .md files, downloads their content, and tracks ETags for change detection
  2. SyncService: Detects new, modified, or deleted files and performs incremental synchronization (no unnecessary reprocessing)
  3. VectorStore:
    • Splits documents into chunks (1000 characters by default)
    • Generates embeddings using your chosen provider:
      • Ollama: nomic-embed-text (local, free)
      • OpenAI: text-embedding-3-small or text-embedding-3-large (cloud, high-accuracy)
    • Indexes vectors using HNSWLib for fast similarity search
  4. MCP Server: Exposes both Tools and Resources via HTTP:
    • Tools: search_documentation, refresh_index, get_full_document for semantic search and actions
    • Resources: resources/list, resources/read for file discovery and direct access

What is HNSWLib?

HNSWLib (Hierarchical Navigable Small World) is a lightweight, in-memory vector search library that's perfect for this use case:

  • ⚡ Fast: Approximate nearest neighbor search in milliseconds
  • 💾 Simple: Stores indices as local files (no database needed)
  • 🪶 Efficient: Low memory footprint, ideal for personal/small-team documentation
  • 🎯 Accurate: High recall with cosine similarity for semantic search

It's the sweet spot between simplicity and performance for RAG applications.

🔐 Security & Authentication

API Key Authentication (Optional)

By default, the server runs in open access mode for easy local development. For shared or remote deployments, you can enable API key authentication:

# Enable authentication
ENABLE_AUTH=true

# Set your API key
MCP_API_KEY=your-secret-key-here

When authentication is enabled:

  • ✅ All endpoints (except /health) require a valid API key
  • ✅ API key can be provided via:
    • Authorization header (recommended): Authorization: Bearer your-secret-key
    • Query parameter: ?api_key=your-secret-key
  • ✅ Invalid or missing keys return HTTP 401 Unauthorized

Usage Examples:

# With Authorization header (recommended)
curl -H "Authorization: Bearer your-secret-key" http://localhost:3000/mcp

# With query parameter
curl "http://localhost:3000/mcp?api_key=your-secret-key"

MCP Client Configuration with API Key:

{
  "mcpServers": {
    "doc": {
      "type": "streamable-http",
      "url": "http://127.0.0.1:3000/mcp",
      "headers": {
        "Authorization": "Bearer your-secret-key"
      },
      "note": "S3 Documentation RAG Server with authentication"
    }
  }
}

💡 Best Practices:

  • Keep authentication disabled for local development
  • Enable it for shared networks or remote deployments
  • Use strong, randomly generated keys (e.g., openssl rand -hex 32)
  • The /health endpoint is always accessible without authentication for monitoring

MCP Tools

search_documentation

{
  "query": "How to configure S3?",
  "max_results": 4
}

Returns relevant document chunks with similarity scores and sources.

refresh_index

{
  "force": false  // default: incremental sync (recommended)
}

Synchronizes the documentation index with S3, detecting new, modified, or deleted files.

Parameters:

  • force (boolean, optional, default: false)
    • false: Incremental sync - Only processes changes (fast, efficient) ✅
    • true: Full reindex - Reprocesses ALL files (slow, expensive) ⚠️

⚠️ Important: The force parameter should ONLY be set to true when explicitly needed (e.g., "force reindex", "rebuild everything from scratch"). Full reindex is expensive:

  • Re-downloads all files from S3
  • Regenerates all embeddings
  • Rebuilds the entire vector store

For normal operations, always use incremental sync (default behavior).

get_full_document

{
  "s3_key": "docs/authentification_magique_symfony.md"
}

Retrieves the complete content of a Markdown file from S3 along with metadata:

  • Full S3 key: The document's S3 identifier
  • Complete Markdown content: Entire document (not chunked)
  • Metadata: Size in bytes, last modification date, ETag, chunk count (if indexed)

Use Cases:

  • View the complete document after finding it via search_documentation
  • Export documentation for external use
  • Understand the full context around a search result
  • Display complete documents in third-party integrations

Important Notes:

  • If a document appears in search results but get_full_document returns "not found", it means the file was deleted from S3 after being indexed
  • Solution: Run refresh_index to synchronize the index with the current S3 state
  • The tool will provide a helpful error message indicating when a sync is needed

MCP Resources

In addition to the 3 tools, the server implements MCP Resources for file discovery and direct access:

  • resources/list: Lists all indexed Markdown files with metadata (name, URI, size, chunks, last modified)
  • resources/read: Reads the full content of a specific file by its URI (e.g., s3doc://docs/authentication.md)

Use case: When users ask "What files do you have?" or "Show me file X", the LLM can browse and access files directly without semantic search.

🤝 Contributing

Contributions are welcome! Please read our Contributing Guide for details on how to submit pull requests, report issues, and contribute to the project.

📝 License

MIT

👤 Author

Yoan Bernabeu