scrape_urlLABSCO SUMMARY
Search and scrape in one call is the one that changes how you ask.
Scraping a URL you already have is the easy case; search_and_scrape means you can ask about something without knowing where the answer lives, and the model gets the pages rather than a list of links. Batch mode returning per-URL errors instead of failing wholesale is what makes a ten-page comparison survive one bad site.
What it is
The CrawlAPI SDK repository, which ships an MCP server alongside the JavaScript client. Point it at a URL and you get back markdown a model can read, with full JavaScript rendering — so pages that build themselves in the browser come back as content rather than an empty shell.
What you get
scrape_urlfetches one page and returns clean markdownbatch_scrapehandles up to 10 URLs in parallel; a failed URL returns an error field instead of failing the whole requestsearch_and_scrapesearches the web and scrapes the top results in one call- Formats beyond markdown are available — html, text and structured — along with
waitForfor JavaScript-heavy pages and atimeoutdefaulting to 30000 ms with a 60s maximum - Scrape results carry metadata such as the page title
Requirements
Node.js 18 or newer and a RapidAPI key — the free tier is 50 calls a day with no card. Clone the repository and run npm install, then point your client at node with the absolute path to mcp-server.js and CRAWLAPI_KEY in the environment. CRAWLAPI_BASE_URL overrides the API base if you need it. The same config works in Cursor, Windsurf and Continue; the server uses stdio transport.
Setup effort
Build from source — git clone https://github.com/JDWolf96/crawlapi-js
USE CASES
01
Reading a page the model can't otherwise see
Fetch it as markdown, JavaScript rendering included.
02
Comparing several competitor pages
Batch them in one request and let the failures come back individually.
03
Researching something you don't have links for
Search and scrape the top results in a single call.
04
Getting content out of a page that loads late
Give it a wait before capture rather than accepting an empty shell.
3 tools
Listed in project docs. Your client may show a different list.
batch_scrapesearch_and_scrapeBEFORE YOU INSTALL
What to know first
Tools you get
3 tools named in the project's documentation.
Requires
CRAWLAPI_KEY in your environment
Where it runs
All 4 documented clients run the server on your machine.
INSTALLATION
Connect your client
Server setup
TERMINAL
git clone https://github.com/JDWolf96/crawlapi-js
cd crawlapi-js
npm installMCP Clients
Claude Desktop, Cursor, Windsurf, Continue
JSON
{
"mcpServers": {
"crawlapi": {
"command": "node",
"args": [
"<YOUR_PATH_TO_CRAWLAPI_JS>/mcp-server.js"
],
"env": {
"CRAWLAPI_KEY": "<YOUR_CRAWLAPI_KEY>"
}
}
}
}
