LabscoMCP early access ↗
bneilMCP SERVER · OFFICIAL PROJECT

MCP Go Colly Crawler

Crawl one URL or a batch of them with a configurable depth, and get the extracted content back — domain restrictions applied automatically from the URLs you gave.

by bneil ↗Site Crawling & Scraping Infrastructure✓ Official

LABSCO SUMMARY

Batch and depth in one call, scoped to where you pointed it.

Most fetch tools give an assistant one page at a time, so gathering a documentation section becomes a dozen round trips. Taking a list of URLs and a depth in a single call collapses that, and deriving the domain restriction from the URLs themselves removes the usual configuration step while quietly preventing a crawl that escapes onto the wider web. The default depth of 2 is a sensible line: enough to follow a page's own links, not enough to accidentally pull a whole site.

What it is

A web crawler exposed over MCP, built on Colly and written in Go. It takes one URL or a list, follows links to a depth you choose, and returns the extracted content in a form an assistant can use — with crawling restricted to the domains of the URLs you supplied, so a crawl cannot wander off the site you asked about.

What you get

  • Concurrent crawling of one or several URLs in a single call, passed as urls
  • Depth you control with max_depth, defaulting to 2 — so a page and its immediate links, unless you say otherwise
  • Domain restrictions applied automatically from the URLs you provided, with no allowlist to maintain
  • Results formatted for a language model to read, rather than raw HTML
  • Graceful shutdown and error handling around the crawl, so a failure returns something rather than hanging

Requirements

No account and no key. Go 1.21 or later and Make to build: make deps then make build, which puts the binary in bin/mcp-go-colly; make build-all cross-compiles for Linux, Windows and macOS. The client entry points its command at the built binary — no runtime dependencies beyond it. MIT licensed.

Setup effort

Build from source — git clone https://github.com/bneil/mcp-go-colly.git

See installation options