The asynchronous pair is the shape to plan around: create_crawl and batch_scrape_urls hand back an id straight away, and the pages only arrive through get_crawl_results or get_batch_results — the crawl side asks to be polled about every 10 seconds. Several tools cover overlapping ground: create_map and get_website_urls both discover links without scraping, and scrape_website and get_webpage_content both fetch a single page. answers draws its own boundary — good for a bounded fact such as a founding year or a current price, and by its own description not reliable for enumerating a live list from a page.
A client-side front end to Olostep's scraping API, splitting the work into URL discovery, whole-site crawling, batch scraping of a known list, single-page extraction, and search.
- URL discovery on a site — the full link list, or one filtered by a search query — returned without scraping any content
- A whole-domain crawl started from one URL with a page cap, country and output format, then polled by crawl_id until the pages and their content come back
- A specific list of URLs from different domains scraped asynchronously and collected by batch_id
- Single-page extraction with JavaScript rendering, a wait before scraping, a choice of parser and output format, and optional LLM extraction
- Parser-based web search returning structured results, and a bounded factual question answered with sources and citations, optionally shaped into a flat JSON object of fields you name
An Olostep API key in OLOSTEP_API_KEY.
One command plus a key — env OLOSTEP_API_KEY=your-api-key npx -y olostep-mcp, then supply credentials