Summary
A scraper that gets past the overlays instead of returning them.
Most page-to-Markdown tools hand you the cookie wall. This one screenshots the page, asks a vision model what to click, and repeats until the content is visible — which is why it needs a model key that plain scrapers do not. Every scrape costs a few vision calls, so it earns its place on pages a simple fetch cannot reach.
What it is
A single-tool scraper: scrape-webpage drives Puppeteer in stealth mode, extracts the main content with Mozilla's Readability, and converts it to Markdown with Turndown.
What you get
- Page content as Markdown, with code blocks and tables handled rather than flattened
- A vision model looks at screenshots of the page and decides what to click, so cookie consent, CAPTCHAs, newsletter prompts, paywalls, age gates and interstitial ads get cleared before extraction
- Automatic interaction can be switched off per call, and the number of attempts capped
- An option to wait for the network to go quiet before reading the page
- An option to append the page's same-domain links to the Markdown output
- Headless by default, with a visible-browser mode for watching what it does
- Results carry a status message and the size of the extracted content, so a silent failure reads as a failure
- Stdio, SSE or streamable HTTP transport
Requirements
A key for a vision-capable model — an OpenAI model by default, with any OpenAI-compatible endpoint usable instead. Node and npm; the browser runs locally.
Setup effort
One command plus a key — npx -y puppeteer-vision-mcp-server, then supply credentials