firecrawl-crawl

by firecrawl

Bulk extract content from entire websites or site sections with depth and path filtering. Crawls pages following links up to configurable depth limits and page counts, with path inclusion/exclusion filters to scope extraction Supports async job polling or synchronous waiting with progress display via --wait and --progress flags Offers concurrency control, request delays, and JSON output formatting for integration into agent workflows Part of a four-step escalation pattern: search → scrape →... npx skills add https://github.com/firecrawl/cli --skill firecrawl-crawl Download ZIPGitHub507

firecrawl crawl

Bulk extract content from a website. Crawls pages following links up to a depth/limit.

When to use

• You need content from many pages on a site (e.g., all /docs/)

• You want to extract an entire site section

• Step 4 in the workflow escalation pattern: search → scrape → map → crawl → interact

Options

Option Description --wait Wait for crawl to complete before returning --progress Show progress while waiting --limit <n> Max pages to crawl --max-depth <n> Max link depth to follow --include-paths <paths> Only crawl URLs matching these paths --exclude-paths <paths> Skip URLs matching these paths --delay <ms> Delay between requests --max-concurrency <n> Max parallel crawl workers --pretty Pretty print JSON output -o, --output <path> Output file path

Tips

• Always use --wait when you need the results immediately. Without it, crawl returns a job ID for async polling.

• Use --include-paths to scope the crawl — don't crawl an entire site when you only need one section.

• Crawl consumes credits per page. Check firecrawl credit-usage before large crawls.

See also

• firecrawl-scrape — scrape individual pages

• firecrawl-map — discover URLs before deciding to crawl

• firecrawl-download — download site to local files (uses map + scrape)