list_extraction_contracts
List built-in extraction contracts agents can use for validated schema extraction. Use this before diagnose_extraction_contract or extract_extraction_contract.
LABSCO SUMMARY
The diagnostic mode is the unusual part.
Most scrapers let you find out a page was unextractable after you paid a model to try. Scoring readiness first — with named blockers and a recommended next step, and no LLM call in the loop — turns that into a decision you make before spending. The contract validation follows the same instinct: an agent branching on whether required fields are present is doing something a confidence score cannot support.
What it is
A web scraping engine with an MCP server build. Three modes sit on the same crawler: a plain crawl that returns rich page data, an LLM extraction mode driven by your JSON Schema, and a diagnostic mode that scores whether URLs are ready for extraction without making an LLM call at all.
What you get
- Rich page data per URL: title, description, language, canonical URL, robots directives, full text, boilerplate-stripped markdown, links flagged internal or external, images with their lazy-load source, meta tags, OG and Twitter Card, JSON-LD, microdata, commerce data, forms with field types, hreflang, pagination, redirect chain, and response headers with timing
- Extraction against a schema you supply, pointed at any OpenAI-compatible chat-completions endpoint — your own vLLM, llama.cpp, LM Studio or Ollama, or a hosted one — with a natural-language prompt usable instead of or alongside the schema
- Validated extraction contracts, so a result carries
validation.validand the list of missing required fields rather than loose markdown an agent has to trust — the built-in contracts arereal-estate-listing,product-pageanddocs-page - A readiness pass with no LLM in it:
diagnoseModereturns a per-URL verdict, a readiness flag, a score, blockers, warnings and a recommended next step, anddiagnosticReportsaves a readable Markdown summary - Errors an agent can branch on instead of regex over strings: an
errorTypeenum covering dns, timeout, rate-limit, blocked-bot, js-required, http-4xx, http-5xx, parse, network and unknown, paired with anerrorRetryableboolean - Challenge-page detection, so a 200 response whose body is an access-control interstitial is reported as blocked rather than returned as content
- Content controls that cut a page down before it costs anything:
onlyMainContentplus CSS allow and deny lists across text, markdown, links and HTML output, with cleaned and raw HTML both available - Brand identity extraction with no LLM: one call returns a ranked colour palette, theme colour and best-guess logo candidates, reading rendered colours in browser mode so it works on SPAs where static CSS does not
- Adaptive crawling that starts with a fast HTTP parse and escalates to a browser only when an SPA shell is detected, plus heading-aware chunking at h1–h3 boundaries with overlap and a per-chunk hash for feeding a vector database
- Search and sitemap entry points: top-N results for a query, or a sitemap URL resolved through sitemap-index files, and PDF and DOCX URLs auto-detected and parsed
Requirements
For the MCP build: clone the repository, build the engine, and run the MCP entry point from the build output — the README notes the published npm package lags the GitHub source, so the current contracts and MCP tools come from the source path. Node.js. Extraction mode needs a reachable OpenAI-compatible endpoint and its key, which should be supplied as an environment variable rather than in run inputs; on the hosted Apify actor that endpoint must be publicly reachable, because LAN addresses such as http://192.168.x.x cannot be reached from Apify's infrastructure. Running it yourself avoids the hosted per-page charge; on Apify the crawl is $0.005 per successfully scraped page, with failed pages not charged, and dryRun gives you a smoke test that emits no billing event. Licence is AGPL-3.0.
Setup effort
Build from source — git clone https://github.com/manchittlab/TheCrawler.git
USE CASES
01
Deciding whether a scraping job is worth building
Run the readiness pass over a sample of real URLs and read the verdicts, blockers and recommended next step before writing any extraction code.
02
Pulling structured records off pages that were not designed for it
Define the fields you need as a schema, point the extractor at the endpoint you already run, and branch on whether the required fields came back.
03
Feeding a knowledge base
Crawl to boilerplate-stripped markdown, chunk it at heading boundaries with overlap, and send the chunks and their hashes to a vector database.
04
Handling failures programmatically
Branch on the error type and the retryable flag — retrying a timeout, escalating a bot block, and not retrying a parse failure — instead of matching on error text.
05
Lifting a site's visual identity
Read the ranked palette, theme colour and logo candidates from the rendered page, with no model call involved.
8 tools
Listed in project code.
diagnose_extraction_contract
Run a no-LLM readiness diagnostic for one extraction contract before spending LLM tokens. Returns per-URL verdicts, scores, blockers, branchable error types, missing readiness signals, and a workflow-level recommended next step. This does not claim a site will work; it tells the agent whether to extract, retry with browser/proxy, review manually, or stop.
extract_extraction_contract
Crawl URLs and extract against a built-in validated extraction contract with an OpenAI-compatible LLM. Returns parsed data plus required-field validation and missingRequiredFields evidence. Use diagnose_extraction_contract first when the source is unknown.
crawl
Scrape one or more URLs. Returns rich structured data: title, description, text, markdown, links, images, meta, OG/Twitter cards, JSON-LD, microdata, headings, tables, forms, redirect chain, hreflang, pagination links, commerce data (price/rating/sku from JSON-LD), analytics tracker detection (16 trackers), and optional email-like/phone-like public text fields when explicitly enabled. Auto-handles PDF and DOCX URLs. Default Cheerio (fast HTTP+parse); set usePlaywright=true for JS rendering, or adaptiveCrawling=true to auto-detect SPAs and re-crawl them with Playwright. On failure, returns structured error with errorType (dns|timeout|rate-limit|blocked-bot|js-required|http-4xx|http-5xx|parse|network|unknown) and retryable flag.
crawl_markdown
Extract clean markdown for a single URL. Strips boilerplate (nav/header/footer/cookie banners). Returns markdown text only — no surrounding metadata. Ideal for feeding a single page to an LLM. For multi-URL or structured output, use crawl, extract_structured, or extract_extraction_contract instead.
search_and_crawl
Search Google for a query, then scrape the top N results. Returns structured data per result page. Without a SerpAPI key, falls back to scraping Google's HTML — fragile, can break when Google updates their HTML. For production agent use, supply serpApiKey.
crawl_sitemap
Fetch URL list from a sitemap.xml (supports sitemap-index files), then optionally scrape them. Use scrape=false to just get the URL list (cheap), scrape=true to also extract content from each.
extract_structured
LLM-powered structured extraction. Crawls each URL → cleans to markdown → asks an OpenAI-compatible LLM (llama.cpp / vLLM / LM Studio / Ollama / OpenAI) to return ONLY a JSON object matching your jsonSchema or prompt. Returns parsed typed data, not raw markdown. LLM endpoint configurable per-call or via env vars (THECRAWLER_LLM_BASEURL, THECRAWLER_LLM_MODEL, THECRAWLER_LLM_API_KEY).
BEFORE YOU INSTALL
What to know first
Tools you get
8 tools registered in the project's code.
Requires
- Node.js 18 or newer, with npm
- Only for the two extract tools: an OpenAI-compatible chat-completions endpoint, set as THECRAWLER_LLM_BASEURL and THECRAWLER_LLM_MODEL, and its key if the endpoint needs one
Where it runs
All 4 documented clients run the server on your machine.
INSTALLATION
Connect your client
Server setup
TERMINAL
git clone https://github.com/manchittlab/TheCrawler.git
cd TheCrawler/engine
npm install
npm run build
node dist/mcp.jsMCP Clients
Cline, Claude Code, Cursor, Windsurf
JSON
{
"mcpServers": {
"thecrawler": {
"command": "node",
"args": [
"<YOUR_PATH_TO_CRAWLER>/engine/dist/mcp.js"
],
"env": {
"NODE_OPTIONS": "--use-system-ca"
}
}
}
}
