Most extractors drive Chrome, which works and costs hundreds of megabytes of memory and a second or two per page. Matching a browser's TLS fingerprint gets past the fetch-level blocking that motivates the browser in the first place, without the browser — much faster and much lighter, at the cost of pages that genuinely need JavaScript to render. Know which kind of page you are fetching. Also read the licence before embedding it in a product: AGPL is a deliberate choice by the author.
A content extractor for retrieval pipelines. Its distinguishing choice is fetching with browser-grade TLS fingerprinting rather than driving a real browser.
- A page returned as clean markdown, or as structured JSON
- Output shaped for retrieval pipelines rather than for reading
- The same engine available as a command-line tool, a REST API and SDKs, not only as a server
Runs locally as a binary. AGPL-3.0 licensed, which matters if you plan to build a service on top of it.
One command — npx create-webclaw