Summary

Check the known issues before crawling something large.

Two open issues are worth knowing up front: saved files currently flatten the URL structure, so a deep site loses its hierarchy on disk, and there is no document limit, so a very large documentation site has no natural stopping point. Neither matters for a mid-sized plugin's docs, and both matter for something the size of a cloud provider's reference. The categoriser deciding tools-versus-APIs on its own is the part that makes repeated use painless.

What it is

A crawler you drive from your client. Point it at a documentation site and it walks the pages with Playwright, converts them to Markdown with code blocks and tables preserved, works out whether the site is a tool or an API, and files the output under the matching directory. A second tool turns what it downloaded into a condensed reference sheet.

What you get

`crawl_documentation` downloads a site to Markdown, with `max_depth` defaulting to 3 and `force_refresh` for regenerating what is already there
`generate_cheatsheet` condenses a set of docs into a reference sheet, with `use_local` to work from files you already downloaded
`list_documentation` shows what is stored, filtered to `tools`, `apis` or `all`, optionally with statistics
Automatic categorisation: a crawled site lands under `tools/` or `apis/` based on its URL patterns and content, with no per-site configuration
Discovery before crawling — it reports what already exists rather than silently re-downloading
Configurable crawling: `crawler.defaultMaxDepth` 3, `crawler.defaultRateLimit` 2 requests per second, `crawler.pageTimeout` 30000 ms, and `cheatsheet.maxLength` 10000 characters
Rate limiting and robots.txt are respected

Requirements

No account and no key. Node.js 18 or newer. Clone, `npm install`, `npm run build`, then point your client at `node` with `dist/index.js`. Package `mcp-for-docs`, version 0.4.0. Before first use, copy `config.example.json` to `config.json` and set `docsBasePath` — it has no default, and the committed config is tracked in git, so on a second machine either edit it or override with the `DOCS_BASE_PATH` environment variable. The README notes it is tested on macOS.