Labsco
rahulretnan logo

RAG Documentation MCP Server

★ 62

from rahulretnan

Retrieve and process documentation using vector search to provide relevant context for AI assistants.

🔥🔥🔥🔥✓ VerifiedAccount requiredAdvanced setup
#automation#scraping#database#devops#git#maps#monitoring#memory

RAG Documentation MCP Server

smithery badge

An MCP server implementation that provides tools for retrieving and processing documentation through vector search, enabling AI assistants to augment their responses with relevant documentation context.

Table of Contents

Features

Tools

  1. search_documentation

    • Search through the documentation using vector search
    • Returns relevant chunks of documentation with source information

  2. list_sources

    • List all available documentation sources
    • Provides metadata about each source

  3. extract_urls

    • Extract URLs from text and check if they're already in the documentation
    • Useful for preventing duplicate documentation

  4. remove_documentation

    • Remove documentation from a specific source
    • Cleans up outdated or irrelevant documentation

  5. list_queue

    • List all items in the processing queue
    • Shows status of pending documentation processing

  6. run_queue

    • Process all items in the queue
    • Automatically adds new documentation to the vector store

  7. clear_queue

    • Clear all items from the processing queue
    • Useful for resetting the system

  8. add_documentation

    • Add new documentation directly to the system by providing a URL
    • Automatically fetches, processes, and indexes the content
    • Supports various web page formats and extracts relevant content
    • Chunks content intelligently for optimal retrieval
    • Required parameter: url (must include protocol, e.g., https://)

  9. add_repository

    • Index a local code repository for documentation
    • Configure include/exclude patterns for files and directories
    • Supports different chunking strategies based on file types
    • Uses asynchronous processing to avoid MCP timeouts with large repositories
    • Provides detailed progress logging (heartbeat) to stderr during indexing
    • Required parameter: path (absolute path to repository)

  10. list_repositories

    • List all indexed repositories with their configurations
    • Shows include/exclude patterns and watch status

  11. update_repository

    • Re-index a repository with updated configuration
    • Can modify include/exclude patterns and other settings
    • Provides detailed progress logging (heartbeat) to stderr during re-indexing
    • Required parameter: name (repository name)

  12. remove_repository

    • Remove a repository from the index
    • Deletes all associated documents from the vector database
    • Required parameter: name (repository name)

  13. watch_repository

    • Start or stop watching a repository for changes
    • Automatically updates the index when files change
    • Required parameters: name (repository name) and action ("start" or "stop")

  14. get_indexing_status

    • Get the current status of repository indexing operations
    • Provides detailed information about ongoing or completed indexing processes
    • Shows progress percentage, file counts, and timing information
    • Optional parameter: name (repository name) - if not provided, returns status for all repositories

Web Interface

The system includes a web interface that can be accessed after starting the Docker Compose services:

  1. Open your browser and navigate to: http://localhost:3030
  2. The interface provides:
    • Real-time queue monitoring
    • Documentation source management
    • Search interface for testing queries
    • System status and health checks

Documentation Management

Direct vs. Queue-Based Documentation Addition

The system provides two complementary approaches for adding documentation:

  1. Direct Addition (add_documentation tool)

    • Immediately processes and indexes the documentation from a URL
    • Best for adding individual documentation sources
    • Provides immediate feedback on processing success/failure
    • Example usage: add_documentation with url: "https://example.com/docs"

  2. Queue-Based Processing

    • Add URLs to a processing queue (extract_urls with add_to_queue: true)
    • Process multiple URLs in batch later (run_queue)
    • Better for large-scale documentation ingestion
    • Allows for scheduled processing of many documentation sources
    • Provides resilience through the queue system

Choose the approach that best fits your documentation management needs. For small numbers of important documents, direct addition provides immediate results. For large documentation sets or recursive crawling, the queue-based approach offers better scalability.

Local Repository Indexing

The system supports indexing local code repositories, making their content searchable alongside web documentation:

  1. Repository Configuration

    • Define which files to include/exclude using glob patterns
    • Configure chunking strategies per file type
    • Set up automatic change detection with watch mode

  2. File Processing

    • Files are processed based on their type and language
    • Code is chunked intelligently to preserve context
    • Metadata like file path and language are preserved

  3. Asynchronous Processing

    • Large repositories are processed asynchronously to avoid MCP timeouts
    • Indexing continues in the background after the initial response
    • Progress can be monitored using the get_indexing_status tool
    • Smaller batch sizes (50 chunks per batch) improve responsiveness

  4. Change Detection

    • Repositories can be watched for changes
    • Modified files are automatically re-indexed
    • Deleted files are removed from the index

Example usage:

add_repository with {
  "path": "/path/to/your/repo",
  "name": "my-project",
  "include": ["**/*.js", "**/*.ts", "**/*.md"],
  "exclude": ["**/node_modules/**", "**/dist/**"],
  "watchMode": true
}

After starting the indexing process, you can check its status:

get_indexing_status with {
  "name": "my-project"
}

This will return detailed information about the indexing progress:

Repository: my-project
Status: 🔄 Processing
Progress: 45%
Started: 5/11/2025, 2:45:30 PM
Duration: 3m 15s
Files: 120 processed, 15 skipped (of 250)
Chunks: 1500 indexed (of 3300)
Batch: 15 of 33

Repository Configuration File

The system supports a repositories.json configuration file that allows you to define repositories to be automatically indexed at startup:

{
  "repositories": [
    {
      "path": "/path/to/your/repo",

The configuration file is automatically updated when repositories are added, updated, or removed using the repository management tools. You can also manually edit the file to configure repositories before starting the server. The paths within the configuration file, such as the path for each repository and the implicit location of repositories.json itself, are resolved relative to the project root directory where the server is executed.

Configuration Options:

  • repositories: Array of repository configurations
    • path: Absolute path to the repository directory "name": "my-project", "include": ["/*.js", "/.ts", "**/.md"], "exclude": ["/node_modules/", "/.git/"], "watchMode": true, "watchInterval": 60000, "chunkSize": 1000, "fileTypeConfig": { ".js": { "include": true, "chunkStrategy": "semantic" }, ".ts": { "include": true, "chunkStrategy": "semantic" }, ".md": { "include": true, "chunkStrategy": "semantic" } } } ], "autoWatch": true }

The configuration file is automatically updated when repositories are added, updated, or removed using the repository management tools. You can also manually edit the file to configure repositories before starting the server.

**Configuration Options:**

- `repositories`: Array of repository configurations
  - `path`: Absolute path to the repository directory
  - `name`: Unique name for the repository
  - `include`: Array of glob patterns to include
  - `exclude`: Array of glob patterns to exclude
  - `watchMode`: Whether to watch for changes
  - `watchInterval`: Polling interval in milliseconds
  - `chunkSize`: Default chunk size for files
  - `fileTypeConfig`: Configuration for specific file types
    - `include`: Whether to include this file type
    - `chunkStrategy`: Chunking strategy ("semantic", "line", or "character")
    - `chunkSize`: Optional override for chunk size

- `autoWatch`: Whether to automatically start watching repositories with `watchMode: true` at startup

## Acknowledgments

This project is a fork of [qpd-v/mcp-ragdocs](https://github.com/qpd-v/mcp-ragdocs), originally developed by qpd-v. The original project provided the foundation for this implementation.

Special thanks to the original creator, qpd-v, for their innovative work on the initial version of this MCP server. This fork has been enhanced with additional features and improvements by Rahul Retnan.