crawl_documentationLABSCO SUMMARY
Check the known issues before crawling something large.
Two open issues are worth knowing up front: saved files currently flatten the URL structure, so a deep site loses its hierarchy on disk, and there is no document limit, so a very large documentation site has no natural stopping point. Neither matters for a mid-sized plugin's docs, and both matter for something the size of a cloud provider's reference. The categoriser deciding tools-versus-APIs on its own is the part that makes repeated use painless.
What it is
A crawler you drive from your client. Point it at a documentation site and it walks the pages with Playwright, converts them to Markdown with code blocks and tables preserved, works out whether the site is a tool or an API, and files the output under the matching directory. A second tool turns what it downloaded into a condensed reference sheet.
What you get
crawl_documentationdownloads a site to Markdown, withmax_depthdefaulting to 3 andforce_refreshfor regenerating what is already theregenerate_cheatsheetcondenses a set of docs into a reference sheet, withuse_localto work from files you already downloadedlist_documentationshows what is stored, filtered totools,apisorall, optionally with statistics- Automatic categorisation: a crawled site lands under
tools/orapis/based on its URL patterns and content, with no per-site configuration - Discovery before crawling — it reports what already exists rather than silently re-downloading
- Configurable crawling:
crawler.defaultMaxDepth3,crawler.defaultRateLimit2 requests per second,crawler.pageTimeout30000 ms, andcheatsheet.maxLength10000 characters - Rate limiting and robots.txt are respected
Requirements
No account and no key. Node.js 18 or newer. Clone, npm install, npm run build, then point your client at node with dist/index.js. Package mcp-for-docs, version 0.4.0. Before first use, copy config.example.json to config.json and set docsBasePath — it has no default, and the committed config is tracked in git, so on a second machine either edit it or override with the DOCS_BASE_PATH environment variable. The README notes it is tested on macOS.
Setup effort
Build from source — git clone https://github.com/shayonpal/mcp-for-docs.git
USE CASES
01
Getting a library's docs available offline
Crawl the site once and query the Markdown locally instead of fetching pages every session.
02
Producing a quick reference from long docs
Condense what you downloaded into a cheat sheet rather than re-reading the whole site.
03
Checking what you already have
List the stored documentation with statistics before starting another crawl.
04
Refreshing docs after a major release
Re-crawl with the refresh flag set so existing content is regenerated rather than skipped.
3 tools
Listed in project docs. Your client may show a different list.
generate_cheatsheetlist_documentationBEFORE YOU INSTALL
What to know first
Tools you get
3 tools named in the project's documentation.
Requires
Node.js 18+
Where it runs
Both documented clients run the server on your machine.
INSTALLATION
Connect your client
You clone and build this one yourself.
Server setup
TERMINAL
git clone https://github.com/shayonpal/mcp-for-docs.git
cd mcp-for-docs
npm install
npm run build
cp config.example.json config.jsonEdit config.json and set docsBasePath to the folder where the documentation should be saved (or set the DOCS_BASE_PATH environment variable).
MCP Clients
Claude Desktop, Claude Code
JSON
{
"mcpServers": {
"mcp-for-docs": {
"command": "node",
"args": [
"<YOUR_PATH_TO_MCP_FOR_DOCS>/dist/index.js"
],
"env": {}
}
}
}
