MCP Go Colly Crawler

Overview

MCP Go Colly is a sophisticated web crawling framework that integrates the Model Context Protocol (MCP) with the powerful Colly web scraping library. This project aims to provide a flexible and extensible solution for extracting web content for large language model (LLM) applications.

Features

• Concurrent web crawling with configurable depth and domain restrictions
• MCP server integration for tool-based crawling
• Graceful shutdown handling
• Robust error handling and result formatting
• Support for both single URL and batch URL crawling

Building from Source

Prerequisites

• Go 1.21 or later
• Make (for using Makefile commands)

Installation

1. Clone the repository:

git clone https://github.com/yourusername/mcp-go-colly.git
cd mcp-go-colly

2. Install dependencies:

make deps

Building

The project includes a Makefile with several useful commands:

# Build the binary (outputs to bin/mcp-go-colly)
make build

# Build for all platforms (Linux, Windows, macOS)
make build-all

# Run tests
make test

# Clean build artifacts
make clean

# Format code
make fmt

# Run linter
make lint

All binaries will be generated in the bin/ directory.

Then you need to add the following configuration to the claude_desktop_config.json file:

{
  "mcpServers": {
    "web-scraper": {
      "command": "<add path here>/mcp-go-colly/bin/mcp-go-colly"
    }
  }
}

Contributing

1. Fork the repository
2. Create your feature branch
3. Commit your changes
4. Push to the branch
5. Create a Pull Request

License

MIT

Acknowledgments

• Colly Web Scraping Framework
• Mark3 Labs MCP Project

Usage

As an MCP Tool

The crawler is implemented as an MCP tool that can be called with the following parameters:

{
    "urls": ["https://example.com"],  // Single URL or array of URLs
    "max_depth": 2                    // Optional: Maximum crawl depth (default: 2)
}

Example MCP Tool Call

result, err := crawlerTool.Call(ctx, mcp.CallToolRequest{
    Params: struct{ Arguments map[string]interface{} }{
        Arguments: map[string]interface{}{
            "urls": []string{"https://example.com"},
            "max_depth": 2,
        },
    },
})

Configuration Options

• max_depth: Set maximum crawl depth (default: 2)
• urls: Single URL string or array of URLs to crawl
• Domain restrictions are automatically applied based on the provided URLs