Most scrapers let you find out a page was unextractable after you paid a model to try. Scoring readiness first — with named blockers and a recommended next step, and no LLM call in the loop — turns that into a decision you make before spending. The contract validation follows the same instinct: an agent branching on whether required fields are present is doing something a confidence score cannot support.
A web scraping engine with an MCP server build. Three modes sit on the same crawler: a plain crawl that returns rich page data, an LLM extraction mode driven by your JSON Schema, and a diagnostic mode that scores whether URLs are ready for extraction without making an LLM call at all.
- Rich page data per URL: title, description, language, canonical URL, robots directives, full text, boilerplate-stripped markdown, links flagged internal or external, images with their lazy-load source, meta tags, OG and Twitter Card, JSON-LD, microdata, commerce data, forms with field types, hreflang, pagination, redirect chain, and response headers with timing
- Extraction against a schema you supply, pointed at any OpenAI-compatible chat-completions endpoint — your own vLLM, llama.cpp, LM Studio or Ollama, or a hosted one — with a natural-language prompt usable instead of or alongside the schema
- Validated extraction contracts, so a result carries `validation.valid` and the list of missing required fields rather than loose markdown an agent has to trust — the built-in contracts are `real-estate-listing`, `product-page` and `docs-page`
- A readiness pass with no LLM in it: `diagnoseMode` returns a per-URL verdict, a readiness flag, a score, blockers, warnings and a recommended next step, and `diagnosticReport` saves a readable Markdown summary
- Errors an agent can branch on instead of regex over strings: an `errorType` enum covering dns, timeout, rate-limit, blocked-bot, js-required, http-4xx, http-5xx, parse, network and unknown, paired with an `errorRetryable` boolean
- Challenge-page detection, so a 200 response whose body is an access-control interstitial is reported as blocked rather than returned as content
- Content controls that cut a page down before it costs anything: `onlyMainContent` plus CSS allow and deny lists across text, markdown, links and HTML output, with cleaned and raw HTML both available
- Brand identity extraction with no LLM: one call returns a ranked colour palette, theme colour and best-guess logo candidates, reading rendered colours in browser mode so it works on SPAs where static CSS does not
- Adaptive crawling that starts with a fast HTTP parse and escalates to a browser only when an SPA shell is detected, plus heading-aware chunking at h1–h3 boundaries with overlap and a per-chunk hash for feeding a vector database
- Search and sitemap entry points: top-N results for a query, or a sitemap URL resolved through sitemap-index files, and PDF and DOCX URLs auto-detected and parsed
For the MCP build: clone the repository, build the engine, and run the MCP entry point from the build output — the README notes the published npm package lags the GitHub source, so the current contracts and MCP tools come from the source path. Node.js. Extraction mode needs a reachable OpenAI-compatible endpoint and its key, which should be supplied as an environment variable rather than in run inputs; the note that LAN addresses are unreachable applies to the hosted actor path. Running it yourself avoids the hosted per-page charge; on Apify the crawl is $0.005 per successfully scraped page, with failed pages not charged, and `dryRun` gives you a smoke test that emits no billing event. Licence is AGPL-3.0.
One command — npm install thecrawler