Scraping four pages puts four pages in front of the model; asking for the specific fact across those four URLs puts back an answer with optional source references. The depth and page limits on crawling exist for the same reason — an unbounded crawl of a documentation site will exhaust a context window long before it exhausts the site.
A Firecrawl-backed research server. Beyond search and scraping, its useful piece is extraction: you give it a list of URLs and a plain-language instruction, and it returns the information you asked for rather than the page you would have had to read.
- A web search from a `query`, returning results as JSON
- Information extracted from a list of `urls` according to a `prompt`, with `enableWebSearch` to supplement from the wider web and `showSources` to include references
- A site crawled from a starting `url` with `maxDepth` and a page `limit`, returning content as markdown or HTML
- A single page scraped from its `url`, with screenshots available
A Firecrawl API key in `FIRECRAWL_API_KEY`, which is the one that must be set. `OPENAI_API_KEY` is optional and powers the AI-assisted features, and a Tavily key is listed as an optional extra search path. Python 3.8 or higher with the `uv` package manager: clone the repository, create the environment with `uv venv`, install with `uv sync`, and have your client launch it as `uv --directory <absolute path> run main.py`. The package is `webtools` (0.1.0).