scan_website_botsLABSCO SUMMARY
Reverse DNS is what unmasks a fake Googlebot.
User-Agent strings are free to forge, so the verification step — PTR lookup, suffix check, forward resolve — is the only part that settles the question. The behavioural scoring on top uses your own access logs, so the recommendation reflects what a bot did on your site rather than its reputation elsewhere.
What it is
An MCP server for bot governance, built around a curated signature database of 58 well-known bots — Googlebot, GPTBot, ClaudeBot, PerplexityBot, Bytespider, AhrefsBot and the rest — each tagged with category, operator, baseline risk and the reverse-DNS suffixes that prove identity. It is polite by design: it respects robots.txt, rate-limits its own fetches and advertises an honest User-Agent.
What you get
- Bot governance (8 tools) —
scan_website_bots,analyze_access_logs,verify_bot_identity,list_allowed_bots,list_blocked_bots,generate_robots_txt,suggest_bot_policy,export_bot_report - Risk scoring that combines UA matching, robots.txt compliance, error rate, request rate and unique-path fan-out into a 0–100 score and one of allow, monitor, rate-limit, block or verify-identity
- Identity verification by PTR plus forward DNS — a spoofed Googlebot comes back as
spoofed - Page content and GEO diagnostics —
fetch_page_content,crawl_pages,check_ai_extractabilityto catch pages that return HTTP 200 but read as blank to an extractor, andvalidate_sitemapsto expand a sitemap index and find children that time out or 5xx generate_llms_txtandcheck_llms_txtto build a grounding corpus and compare it against the live file- Reports in Markdown, JSON and CSV
- A further catalog of website-mirror tools, 88 in total, covering blog, help centre, pricing, features, integrations and glossary content
Requirements
Node, and either the npm package mcp-bot-crawler or a build from source. It speaks stdio, so any MCP client can launch it. DEFAULT_TARGET_URL sets the site to scan and DEFAULT_ACCESS_LOG the log to analyse; the repository ships no sample access log, so point that variable at a real one or pass logPath / logText explicitly. Politeness knobs: CRAWL_DELAY_MS (default 1 s), MAX_SITEMAP_PAGES (default 25) and HTTP_TIMEOUT_MS (default 10 s).
Setup effort
Build from source — git clone https://github.com/MeruLocal/HelloGrowthCRMwebsite_MCP.git
USE CASES
01
Finding out who is actually crawling you
Scan the site and analyse the access log, then read the risk score and recommended action per bot.
02
Checking whether a crawler is who it says it is
Run reverse DNS against the documented suffix and see whether it comes back verified or spoofed.
03
Deciding what to allow AI crawlers
Get a suggested policy, then generate the robots.txt that expresses it.
04
A page ranks but AI answers ignore it
Measure readable text inside <main> against <body> to catch pages that return 200 and extract as blank.
88 tools
Listed in project docs. Your client may show a different list.
analyze_access_logsverify_bot_identitylist_allowed_botslist_blocked_botsgenerate_robots_txtsuggest_bot_policyexport_bot_reportfetch_page_contentcrawl_pagesgenerate_llms_txtcheck_llms_txtcheck_ai_extractabilityvalidate_sitemapsblog_listblog_getblog_searchblog_createblog_updateblog_revalidateblog_get_categorieshelp_list_categorieshelp_list_articleshelp_get_articlehelp_searchhelp_create_articlehelp_update_articlenewsletter_subscribenewsletter_unsubscribenewsletter_get_subscribersnewsletter_get_statsforms_submitforms_list_submissionsforms_get_submissionforms_export_csvcontent_list_case_studiescontent_list_comparisonscontent_get_comparisoncontent_list_industriescontent_list_toolscontent_get_seo_rulespricing_get_planspricing_get_addonspricing_get_faqpricing_compare_planspricing_get_country_planspricing_get_managed_revopsfeatures_listfeatures_getfeatures_list_productsanalytics_social_proofcountries_listcountry_getcompany_get_profilecompany_get_contactsseo_get_site_configseo_get_hreflangseo_get_canonicalseo_get_sitemapsseo_get_schemaproducts_listproduct_getintegrations_listintegrations_getintegrations_list_categoriesagents_listagents_getagents_get_autonomy_levelsagents_list_comparisonsglossary_list_termsglossary_get_termtemplates_listtemplates_getguides_listguides_getalternatives_listalternatives_getswitch_list_competitorsswitch_get_guidechangelog_list_releaseschangelog_get_releasefaqs_get_sitemedia_list_videosmedia_list_testimonialspartners_get_programpartners_get_application_schemasolutions_list_whatsapp_use_casessolutions_get_managed_revopsBEFORE YOU INSTALL
What to know first
Tools you get
88 tools named in the project's documentation.
Optional
SUPABASE_SERVICE_ROLE_KEY in your environment
Where it runs
Of the 3 documented clients, 2 run the server on your machine and 1 connects to a hosted endpoint.
INSTALLATION
Connect your client
Server setup
From source
TERMINAL
git clone https://github.com/MeruLocal/HelloGrowthCRMwebsite_MCP.git
cd HelloGrowthCRMwebsite_MCP
cp .env.example .env # already pre-configured for hellogrowthcrm.com
npm install
npm run buildHosted endpoint
ENDPOINT URL
https://mcp.hellogrowthcrm.com/sseAdd this hosted endpoint to your MCP client.
MCP Clients
Claude Desktop, Claude Code
JSON
{
"mcpServers": {
"bot-crawler": {
"command": "node",
"args": [
"<YOUR_PATH_TO_HELLO_GROWTH_CR_MWEBSITE>/dist/index.js"
],
"env": {
"DEFAULT_TARGET_URL": "https://hellogrowthcrm.com",
"DEFAULT_ACCESS_LOG": "/var/log/nginx/access.log",
"CRAWLER_USER_AGENT": "mcp-bot-crawler/1.0 (+https://hellogrowthcrm.com/bot-info)"
}
}
}
}<YOUR_PATH_TO_HELLO_GROWTH_CR_MWEBSITE> is the full path of the HelloGrowthCRMwebsite_MCP folder you cloned and built. DEFAULT_ACCESS_LOG is the log file to read; /var/log/nginx/access.log is the example. The blog_*, help_*, newsletter_*, forms_* and analytics_social_proof tools also need SUPABASE_URL and SUPABASE_SERVICE_ROLE_KEY in the checkout's .env; every other tool works without them.
Any MCP client — remote
ANY MCP CLIENT — REMOTE · JSON
{
"mcpServers": {
"hellogrowthcrmwebsite-mcp": {
"type": "streamable-http",
"url": "https://mcp.hellogrowthcrm.com/sse"
}
}
}Clients spell the "type" value differently: Claude Code and VS Code use "http", Cline uses "streamableHttp", and Roo Code uses "streamable-http". Claude Desktop does not accept a URL entry in its config file; remote servers go under Settings → Connectors.
