scrape-webpage
Scrapes a webpage and converts it to markdown format
LABSCO SUMMARY
A scraper that gets past the overlays instead of returning them.
Most page-to-Markdown tools hand you the cookie wall. This one screenshots the page, asks a vision model what to click, and repeats until the content is visible — which is why it needs a model key that plain scrapers do not. Every scrape costs a few vision calls, so it earns its place on pages a simple fetch cannot reach.
What it is
A single-tool scraper: scrape-webpage drives Puppeteer in stealth mode, extracts the main content with Mozilla's Readability, and converts it to Markdown with Turndown.
What you get
- Page content as Markdown, with code blocks and tables handled rather than flattened
- A vision model looks at screenshots of the page and decides what to click, so cookie consent, CAPTCHAs, newsletter prompts, paywalls, age gates and interstitial ads get cleared before extraction
- Automatic interaction can be switched off per call, and the number of attempts capped
- An option to wait for the network to go quiet before reading the page
- An option to append the page's same-domain links to the Markdown output
- Headless by default, with a visible-browser mode for watching what it does
- Results carry a status message and the size of the extracted content, so a silent failure reads as a failure
- Stdio, SSE or streamable HTTP transport
Requirements
A key for a vision-capable model — an OpenAI model by default, with any OpenAI-compatible endpoint usable instead. Node and npm; the browser runs locally.
Setup effort
One command plus a key — npx -y puppeteer-vision-mcp-server, then supply credentials
USE CASES
01
Reading an article that hides behind a consent wall
Let the vision loop dismiss the banner and return the article body as Markdown.
02
Getting documentation into the conversation intact
Pull a docs page with its code blocks and tables preserved rather than collapsed into prose.
03
Mapping a section of a site
Ask for the same-domain links alongside the Markdown and you get the page plus where it leads.
04
Debugging a scrape that keeps failing
Run it with the browser visible and watch which overlay it is getting stuck on.
1 tool
Counted once with only startup setup. Other versions, modes or optional credentials may expose a different list.
BEFORE YOU INSTALL
What to know first
Tools you get
1 tool when we connected.
Requires
- npx on PATH
- OPENAI_API_KEY in your environment
Where it runs
All 3 documented setups run the server on your machine.
CONTEXT COST
Light Standing cost
Rated by the size of the one tool definition that rides along on every turn
INSTALLATION
Connect your client
Server setup
TERMINAL
npx -y puppeteer-vision-mcp-serverMCP Clients
Any MCP client — npx
ANY MCP CLIENT — NPX · JSON
{
"mcpServers": {
"web-scraper": {
"command": "npx",
"args": [
"-y",
"puppeteer-vision-mcp-server"
],
"env": {
"OPENAI_API_KEY": "<YOUR_OPENAI_API_KEY>"
}
}
}
}Any MCP client — SSE
ANY MCP CLIENT — SSE · JSON
{
"mcpServers": {
"puppeteer-vision-mcp": {
"url": "http://localhost:3001/sse",
"env": {
"TRANSPORT_TYPE": "sse"
}
}
}
}Any MCP client — HTTP
ANY MCP CLIENT — HTTP · JSON
{
"mcpServers": {
"puppeteer-vision-mcp": {
"url": "http://localhost:3001/mcp",
"env": {
"TRANSPORT_TYPE": "http"
}
}
}
}
