One call runs the whole sequence: cache check, circuit check, stealth browser fetch, backoff retries, plain HTTP fallback, then cache the result. The circuit breaker is what saves a batch job, since a domain that fails 5 times is skipped for the cooldown rather than retried into the ground.
A web-fetching server built for reliability, with a stealth browser fetch, exponential-backoff retries, a plain HTTP fallback, a cache and a per-domain circuit breaker.
- A full reliability fetch returning both markdown and raw HTML (fetch_url)
- Markdown only, for when the HTML is context you do not want to pay for (fetch_markdown)
- Cache hits answered instantly, under a TTL you set
- A per-domain circuit breaker, so a site that is down fails fast instead of consuming retries, with its status readable (check_domain)
- Cache utilisation metrics and a flush when you need fresh data (cache_stats, clear_cache)
- Retries, circuit threshold, cooldown, cache TTL and page timeout all set by environment variable
Pip install steadyfetch and run steadyfetch, or build the Docker image and run it on port 8200. Defaults are 3 retries, a circuit threshold of 5 failures, then a cooldown before it tries again, an hour of caching and a page timeout in the tens of seconds.
One command — pip install steadyfetch