Labsco
JustAzul logo

Web Scraper Service

from JustAzul

A Python-based MCP server for headless web scraping. It extracts the main text content from web pages and outputs it as Markdown, text, or HTML.

🔥🔥🔥✓ VerifiedFreeAdvanced setup
#automation#scraping#browser#documents#cloud#devops#git

Web Scrapper Service (MCP Stdin/Stdout & HTTP)

<!-- Badges: auto-generated, update workflow and Gist as needed -->

Build Test Version License Python PEP8 GHCR Patchright Docker

A Python-based MCP server for robust, headless web scraping—extracts main text content from web pages and outputs Markdown, text, or HTML for seamless AI and automation integration.

Key Features

  • Headless browser scraping (Playwright, BeautifulSoup, Markdownify)
  • Outputs Markdown, text, or HTML
  • Designed for MCP (Model Context Protocol) stdio/JSON-RPC integration
  • Dual transport: stdio (default) and Streamable HTTP for shared service mode
  • Persistent browser pool: Chromium stays alive across requests for fast scraping
  • Smart DOM wait: MutationObserver-based content stabilization instead of fixed sleep
  • Dockerized, with pre-built images
  • Configurable via environment variables
  • Robust error handling (timeouts, HTTP errors, Cloudflare, etc.)
  • Per-domain rate limiting
  • Easy integration with AI tools and IDEs (Cursor, Claude Desktop, Continue, JetBrains, Zed, etc.)
  • One-click install for Cursor, interactive installer for Claude

Transport Modes

stdio (default)

Each MCP client spawns its own container via docker run -i. Simple, zero configuration, works with any MCP client.

Copy & paste — that's it
{
  "mcpServers": {
    "web-scrapper-stdio": {
      "command": "docker",
      "args": ["run", "-i", "--rm", "ghcr.io/justazul/web-scrapper-stdio"]
    }
  }
}

Streamable HTTP (shared service)

Run one persistent container that serves multiple MCP clients over HTTP. Saves resources when running multiple AI tool instances (e.g., multiple Claude Code sessions).

Start the service:

Copy & paste — that's it
docker run -d --name web-scraper \
  -e MCP_TRANSPORT=streamable-http \
  -e MCP_HTTP_PORT=8080 \
  -p 8080:8080 \
  --shm-size=3gb \
  ghcr.io/justazul/web-scrapper-stdio

Connect from your MCP client:

Copy & paste — that's it
{
  "mcpServers": {
    "web-scrapper": {
      "url": "http://localhost:8080/mcp"
    }
  }
}

Integration with AI Tools & IDEs

This service supports integration with a wide range of AI tools and IDEs that implement the Model Context Protocol (MCP). Below are ready-to-use configuration examples for the most popular environments. Replace the image/tag as needed for custom builds.

Cursor IDE

Add to your .cursor/mcp.json (project-level) or ~/.cursor/mcp.json (global):

Copy & paste — that's it
{
  "mcpServers": {
    "web-scrapper-stdio": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "ghcr.io/justazul/web-scrapper-stdio"
      ]
    }
  }
}

Claude Desktop

Add to your Claude Desktop MCP config (typically claude_desktop_config.json):

Copy & paste — that's it
{
  "mcpServers": {
    "web-scrapper-stdio": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "ghcr.io/justazul/web-scrapper-stdio"
      ]
    }
  }
}

Claude Code

Add to your .mcp.json or global ~/.claude.json:

stdio mode (one container per session):

Copy & paste — that's it
{
  "mcpServers": {
    "web-scrapper-stdio": {
      "command": "docker",
      "args": ["run", "-i", "--rm", "ghcr.io/justazul/web-scrapper-stdio"]
    }
  }
}

HTTP mode (shared service — start the service first):

Copy & paste — that's it
{
  "mcpServers": {
    "web-scrapper": {
      "url": "http://localhost:8080/mcp"
    }
  }
}

Continue (VSCode/JetBrains Plugin)

Add to your continue.config.json or via the Continue plugin MCP settings:

Copy & paste — that's it
{
  "mcpServers": {
    "web-scrapper-stdio": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "ghcr.io/justazul/web-scrapper-stdio"
      ]
    }
  }
}

IntelliJ IDEA (JetBrains AI Assistant)

Go to Settings > Tools > AI Assistant > Model Context Protocol (MCP) and add a new server. Use:

Copy & paste — that's it
{
  "command": "docker",
  "args": [
    "run",
    "-i",
    "--rm",
    "ghcr.io/justazul/web-scrapper-stdio"
  ]
}

Zed Editor

Add to your Zed MCP config (see Zed docs for the exact path):

Copy & paste — that's it
{
  "mcpServers": {
    "web-scrapper-stdio": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "ghcr.io/justazul/web-scrapper-stdio"
      ]
    }
  }
}

Development & Testing

Running Tests (Docker Compose)

All tests must be run using Docker Compose. Do not run tests outside Docker.

  • All tests:
    Copy & paste — that's it
    docker compose up --build --abort-on-container-exit test
  • MCP server tests only:
    Copy & paste — that's it
    docker compose up --build --abort-on-container-exit test_mcp
  • Scrapper tests only:
    Copy & paste — that's it
    docker compose up --build --abort-on-container-exit test_scrapper

Running Benchmarks

Copy & paste — that's it
docker compose run --rm benchmark

Results are stored in benchmarks/RESULTS.md.

Contributing

Contributions are welcome! Please open issues or pull requests for bug fixes, features, or improvements. If you plan to make significant changes, open an issue first to discuss your proposal.

License

This project is licensed under the MIT License.