User-Agent strings are free to forge, so the verification step — PTR lookup, suffix check, forward resolve — is the only part that settles the question. The behavioural scoring on top uses your own access logs, so the recommendation reflects what a bot did on your site rather than its reputation elsewhere.
An MCP server for bot governance, built around a curated signature database of 58 well-known bots — Googlebot, GPTBot, ClaudeBot, PerplexityBot, Bytespider, AhrefsBot and the rest — each tagged with category, operator, baseline risk and the reverse-DNS suffixes that prove identity. It is polite by design: it respects robots.txt, rate-limits its own fetches and advertises an honest User-Agent.
- Bot governance (8 tools) — `scan_website_bots`, `analyze_access_logs`, `verify_bot_identity`, `list_allowed_bots`, `list_blocked_bots`, `generate_robots_txt`, `suggest_bot_policy`, `export_bot_report`
- Risk scoring that combines UA matching, robots.txt compliance, error rate, request rate and unique-path fan-out into a 0–100 score and one of allow, monitor, rate-limit, block or verify-identity
- Identity verification by PTR plus forward DNS — a spoofed Googlebot comes back as `spoofed`
- Page content and GEO diagnostics — `fetch_page_content`, `crawl_pages`, `check_ai_extractability` to catch pages that return HTTP 200 but read as blank to an extractor, and `validate_sitemaps` to expand a sitemap index and find children that time out or 5xx
- `generate_llms_txt` and `check_llms_txt` to build a grounding corpus and compare it against the live file
- Reports in Markdown, JSON and CSV
- A further catalog of website-mirror tools, 88 in total, covering blog, help centre, pricing, features, integrations and glossary content
Node, and either the npm package `mcp-bot-crawler` or a build from source. It speaks stdio, so any MCP client can launch it. `DEFAULT_TARGET_URL` sets the site to scan and `DEFAULT_ACCESS_LOG` the log to analyse; the repository ships no sample access log, so point that variable at a real one or pass `logPath` / `logText` explicitly. Politeness knobs: `CRAWL_DELAY_MS` (default 1 s), `MAX_SITEMAP_PAGES` (default 25) and `HTTP_TIMEOUT_MS` (default 10 s).