crawlLABSCO SUMMARY
Submit and collect, rather than wait.
Splitting the crawl into a submit call and a fetch-by-task-id call is the right shape for anything that walks more than one page — the assistant is not blocked while the crawler works, and a result can be picked up later in the conversation. It is a client for a Crawl4ai deployment, not a crawler in itself, so the base URL and token point at infrastructure you run.
What it is
A Spring Boot MCP server that calls the Crawl4ai API. The crawl is asynchronous by design: one call submits the job and another collects the result, so a deep crawl does not hold the tool call open.
What you get
- Crawl submission over an array of target URLs, with a crawl strategy, a maximum depth and an output format —
crawl - Result retrieval for a submitted job by its task id, returned as a JSON string —
task
Requirements
A reachable Crawl4ai server and its credentials, configured as crawl4ai.base-url and crawl4ai.api-token in src/main/resources/application.properties. Java 17 and Maven: clone the repository, and mvn clean install produces target/jcrawl4ai-mcp-server-1.0.0.jar, which your client launches with java -jar over stdio. Logs are written to ./target/mcp-stdio-server.log.
Setup effort
Build from source — git clone https://github.com/rainyheart/jcrawl4ai-mcp-server.git
USE CASES
01
Crawling a documentation site into usable text
Submit the root URLs with a depth and output format, then collect the result once the crawler has worked through them.
02
Running a deep crawl without blocking the conversation
Start the job, carry on, and fetch the result by task id when you need it.
03
Driving an existing Crawl4ai deployment from an assistant
Point the properties file at your Crawl4ai server and token so crawls run on infrastructure you already operate.
2 tools
Listed in project docs. Your client may show a different list.
taskBEFORE YOU INSTALL
What to know first
Tools you get
2 tools named in the project's documentation.
Requires
- Java 17
- Maven
- A Crawl4ai server you can reach, with its API token
Where it runs
The documented client runs the server on your machine.
INSTALLATION
Connect your client
You clone and build this one yourself.
Server setup
TERMINAL
git clone https://github.com/rainyheart/jcrawl4ai-mcp-server.git
cd jcrawl4ai-mcp-serverIn src/main/resources/application.properties, set crawl4ai.base-url and crawl4ai.api-token to your Crawl4ai server's URL and API token
TERMINAL
mvn clean installThe build writes target/jcrawl4ai-mcp-server-1.0.0.jar. In the client configuration, <YOUR_PATH_TO_JAR>/file/ is the full path to that target folder
MCP Clients
Any MCP client
ANY MCP CLIENT · JSON
{
"mcpServers": {
"jcawl4ai-mcp-server": {
"autoApprove": [
"crawl",
"task"
],
"disabled": false,
"timeout": 60,
"command": "java",
"args": [
"-jar",
"<YOUR_PATH_TO_JAR>/file/jcrawl4ai-mcp-server-1.0.0.jar"
],
"transportType": "stdio"
}
}
}
