web_searchLABSCO SUMMARY
Web retrieval that expects to be blocked.
Most fetch tools hand back an error the moment a page throws a bot challenge. Here the challenge is a stage in the pipeline — render it in Chromium, retry through a proxy, or open a real window so you solve it once and the agent carries on alone.
What it is
A single self-hostable Rust binary that gives an agent search, page fetching, structured extraction and stateful browser automation over MCP stdio, with an optional HTTP server on port 5000. When a page resists, it escalates in stages: native retrieval, then Chromium CDP rendering, then a visible browser you drive yourself.
What you get
web_searchfor URL discovery, orweb_search(include_content=true)to search and scrape the top results in one round-tripweb_fetchinsingle,batchorcrawlmode, andextract_fieldsfor structured extractionscout_browser_automate— one tool that takes an array of steps: navigation, hover, click, type, waits, locator assertions, tabs, screenshots and PDF, file upload, form fill, dialog policy, route mocking, console and network capture, and cookie or storage CRUDscout_agent_profile_authopens the agent's own profile in a visible window so you clear a CAPTCHA or a Google login once; the cookies are saved and the agent goes back to running silentlyhitl_web_fetchwithauth_mode="challenge"for CAPTCHA walls orauth_mode="auth"for login walls, andvisual_scoutto judge whether a page needs that firstmemory_searchover a LanceDB-backed history of past research, so a repeat question can skip the live fetchdeep_researchfor multi-hop search, scrape and synthesis against any OpenAI-compatible endpoint, including a local one
Requirements
A prebuilt binary from GitHub Releases, or a source build with protoc on your PATH. The timeout guards belong in the MCP config — they are what stops a stalled page or browser launch from holding the session open. Use RUST_LOG=warn; at info the server floods stderr and confuses clients. memory_search needs LANCEDB_URI pointed at a directory, and deep_research needs an OpenAI-compatible endpoint. MIT.
Setup effort
Follow the published steps
USE CASES
01
Researching a topic you know nothing about yet
One call searches and scrapes the top results together, and the run is remembered so the next question on the same topic can skip the fetching.
02
Pulling the same fields out of a list of pages
Fetch the pages in one batch, then extract the fields you named rather than handing the model whole documents.
03
Getting past a site that keeps blocking you
Check how likely the page is to demand a login, then open a real browser window for the challenge and go back to silent fetching once the cookies are stored.
04
Walking a multi-step flow instead of a single page
Pass one array of steps — navigate, fill the form, click through, assert what should be on screen, capture the network — and the session state survives to the next call.
05
Crawling a documentation site
Crawl mode follows links from a starting page, with the per-page character cap and per-host spacing keeping the output and the request rate in bounds.
17 tools
Listed in project docs. Your client may show a different list.
web_fetchextract_fieldsscout_browser_automatebrowser_automatescout_agent_profile_authscout_browser_closevisual_scouthitl_web_fetchmemory_searchdeep_researchproxy_controlweb_search_jsonweb_fetch_batchweb_crawlfetch_then_extracthuman_auth_sessionBEFORE YOU INSTALL
What to know first
Tools you get
17 tools named in the project's documentation. Six names are aliases: browser_automate plus legacy web_search_json, web_fetch_batch, web_crawl, fetch_then_extract and human_auth_session.
INSTALLATION
Connect your client
Server setup
Option A (prebuilt): download the latest release assets from GitHub Releases — cortex-scout-mcp is the MCP stdio server (recommended for VS Code / Cursor / Claude Desktop); cortex-scout is the optional HTTP server (default port 5000)
Option B step 1 (from source): install protoc first; the release build needs it on your PATH. On macOS:
TERMINAL
brew install protobufOn Ubuntu or Debian:
TERMINAL
sudo apt-get install -y protobuf-compilerOn Fedora:
TERMINAL
sudo dnf install -y protobuf-compilerOption B step 2:
TERMINAL
git clone https://github.com/cortex-works/cortex-scout.git && cd cortex-scoutOption B step 3, basic build (search, scrape, deep research, memory):
TERMINAL
cargo build --release --manifest-path mcp-server/Cargo.toml --bin cortex-scout-mcpOption B step 3 alternative, full build (adds hitl_web_fetch / visible-browser HITL):
TERMINAL
cargo build --release --manifest-path mcp-server/Cargo.toml --all-features --bin cortex-scout-mcpOptional: build the separate HTTP server too:
TERMINAL
cargo build --release --bin cortex-scoutAdd the server entry to your MCP config, pointing the last args element at the built binary; the timeout guard env vars are described as required
MCP Clients
Claude Desktop, Cursor, Windsurf
JSON
{
"mcpServers": {
"cortex-scout": {
"command": "env",
"args": [
"RUST_LOG=warn",
"CORTEX_SCOUT_TOOL_TIMEOUT_SECS=90",
"CORTEX_SCOUT_TOOL_TIMEOUT_SECS_SCRAPE_URL=90",
"CORTEX_SCOUT_TOOL_TIMEOUT_SECS_SEARCH_STRUCTURED=120",
"CORTEX_SCOUT_TOOL_TIMEOUT_SECS_VISUAL_SCOUT=45",
"CORTEX_SCOUT_BROWSER_LAUNCH_TIMEOUT_SECS=12",
"CORTEX_SCOUT_BROWSER_TAB_PROBE_TIMEOUT_SECS=4",
"CORTEX_SCOUT_SCRAPE_STAGE_TIMEOUT_SECS=20",
"CORTEX_SCOUT_SCRAPE_STAGE_TIMEOUT_SECS_CDP_INITIAL_ATTEMPT=25",
"CORTEX_SCOUT_SCRAPE_STAGE_TIMEOUT_SECS_CDP_RETRY_ATTEMPT=25",
"CORTEX_SCOUT_SCRAPE_STAGE_TIMEOUT_SECS_FORCED_CDP_ATTEMPT=25",
"CORTEX_SCOUT_SCRAPE_STAGE_TIMEOUT_SECS_NATIVE_CDP_FALLBACK=25",
"SEARCH_ENGINES=google,bing,duckduckgo,brave",
"LANCEDB_URI=<YOUR_LANCEDB_URI>",
"HTTP_TIMEOUT_SECS=30",
"MAX_CONTENT_CHARS=10000",
"<YOUR_PATH_TO_CORTEX_SCOUT_MCP_SERVER>"
]
}
}
}Continue
Create .continue/mcpServers/cortex-scout.yaml in your project root with this content. MCP tools are only available in agent mode.
CONTINUE · YAML
name: cortex-scout
version: 1.0.0
schema: v1
mcpServers:
- name: cortex-scout
command: env
args:
- RUST_LOG=warn
- CORTEX_SCOUT_TOOL_TIMEOUT_SECS=90
- CORTEX_SCOUT_TOOL_TIMEOUT_SECS_SCRAPE_URL=90
- CORTEX_SCOUT_TOOL_TIMEOUT_SECS_SEARCH_STRUCTURED=120
- CORTEX_SCOUT_TOOL_TIMEOUT_SECS_VISUAL_SCOUT=45
- CORTEX_SCOUT_BROWSER_LAUNCH_TIMEOUT_SECS=12
- CORTEX_SCOUT_BROWSER_TAB_PROBE_TIMEOUT_SECS=4
- CORTEX_SCOUT_SCRAPE_STAGE_TIMEOUT_SECS=20
- CORTEX_SCOUT_SCRAPE_STAGE_TIMEOUT_SECS_CDP_INITIAL_ATTEMPT=25
- CORTEX_SCOUT_SCRAPE_STAGE_TIMEOUT_SECS_CDP_RETRY_ATTEMPT=25
- CORTEX_SCOUT_SCRAPE_STAGE_TIMEOUT_SECS_FORCED_CDP_ATTEMPT=25
- CORTEX_SCOUT_SCRAPE_STAGE_TIMEOUT_SECS_NATIVE_CDP_FALLBACK=25
- SEARCH_ENGINES=google,bing,duckduckgo,brave
- LANCEDB_URI=<YOUR_LANCEDB_URI>
- HTTP_TIMEOUT_SECS=30
- MAX_CONTENT_CHARS=10000
- <YOUR_PATH_TO_CORTEX_SCOUT_MCP_SERVER>
