Labsco
firecrawl logo

firecrawl-crawl

✓ Official★ 507

by firecrawl · part of firecrawl/cli

Bulk extract content from a website. Crawls pages following links up to a depth/limit.

🔥🔥🔥✓ VerifiedFreeQuick setup
#automation#scraping#git
🧰 Not standalone. This skill ships with firecrawl/cli and only works together with that tool — install the tool first, then add this skill.

Bulk extract content from a website. Crawls pages following links up to a depth/limit.

Inspect the full instructions your agent will receiveExpand

This is the exact playbook injected into your agent when the skill activates — shown here so you can audit it before installing. You don't need to read it to use the skill.

by firecrawl

Bulk extract content from a website. Crawls pages following links up to a depth/limit. npx skills add https://github.com/firecrawl/firecrawl-cli --skill firecrawl-crawl Download ZIPGitHub507

firecrawl crawl

Bulk extract content from a website. Crawls pages following links up to a depth/limit.

When to use

  • You need content from many pages on a site (e.g., all /docs/)

  • You want to extract an entire site section

  • Step 4 in the workflow escalation pattern: search → scrape → map → crawl → interact

Options

Option Description --wait Wait for crawl to complete before returning --progress Show progress while waiting --limit <n> Max pages to crawl --max-depth <n> Max link depth to follow --include-paths <paths> Only crawl URLs matching these paths --exclude-paths <paths> Skip URLs matching these paths --delay <ms> Delay between requests --max-concurrency <n> Max parallel crawl workers --pretty Pretty print JSON output -o, --output <path> Output file path

Tips

  • Always use --wait when you need the results immediately. Without it, crawl returns a job ID for async polling.

  • Use --include-paths to scope the crawl — don't crawl an entire site when you only need one section.

  • Crawl consumes credits per page. Check firecrawl credit-usage before large crawls.

See also