Most fetch tools give an assistant one page at a time, so gathering a documentation section becomes a dozen round trips. Taking a list of URLs and a depth in a single call collapses that, and deriving the domain restriction from the URLs themselves removes the usual configuration step while quietly preventing a crawl that escapes onto the wider web. The default depth of 2 is a sensible line: enough to follow a page's own links, not enough to accidentally pull a whole site.
A web crawler exposed over MCP, built on Colly and written in Go. It takes one URL or a list, follows links to a depth you choose, and returns the extracted content in a form an assistant can use — with crawling restricted to the domains of the URLs you supplied, so a crawl cannot wander off the site you asked about.
- Concurrent crawling of one or several URLs in a single call, passed as `urls`
- Depth you control with `max_depth`, defaulting to 2 — so a page and its immediate links, unless you say otherwise
- Domain restrictions applied automatically from the URLs you provided, with no allowlist to maintain
- Results formatted for a language model to read, rather than raw HTML
- Graceful shutdown and error handling around the crawl, so a failure returns something rather than hanging
No account and no key. Go 1.21 or later and Make to build: `make deps` then `make build`, which puts the binary in `bin/mcp-go-colly`; `make build-all` cross-compiles for Linux, Windows and macOS. The client entry points its `command` at the built binary — no runtime dependencies beyond it. MIT licensed.