Having stealth, geolocation and retry as separate tools means the failure has an answer instead of a shrug: blocked becomes stealth, region-locked becomes geolocation, flaky becomes retry with a count. Pairing that with academic search across three databases makes it a research server rather than a scraper — and the content-limit controls matter, because a full crawl will otherwise fill your context.
A research server combining web crawling with academic literature search. The crawling side is built on Crawl4AI and offers several modes for sites that resist a plain fetch; the search side queries scholarly databases directly. The README is in Chinese.
- `academic_search` queries Google Scholar, arXiv or PubMed by passing the source as its second argument
- `crawl` for an ordinary page; `crawl_stealth` for one that blocks automation
- `crawl_with_geolocation` presents a chosen location for region-restricted content
- `crawl_with_retry` takes a `max_retries` count for flaky sites
- `crawl_with_intelligence` picks its approach from the page rather than a fixed mode
- `configure_crawl_settings`, `quick_config_content_limit` and `quick_config_word_threshold` tune how much content comes back
- `system_status` reports whether the server is healthy — the first thing to call after setup
- `experimental_claude_analysis` for content analysis, off unless you configure it
Python 3.12+ with `uv` or pip. After installing dependencies, run `crawl4ai-setup` and install the Chromium browser — the crawler drives a real browser, so this is a heavier install than a fetch-based scraper. Configuration is shown for Amazon Q Developer, pointing at `server_v9.py` with absolute paths. No account or key for crawling and academic search; the Claude analysis tool needs its own API configuration.