inference-server

The model has to be running somewhere before it can answer anything — this starts prime-rl's server for it, on your own machine or as a job on a shared cluster.

WHEN TO USE

SCOPE

USE FOR

Launch the server through prime-rl's own entry point, which sets vLLM's environment up first.
Schedule the inference server as a single-node SLURM job for cluster runs.
Run independent vLLM replicas across multiple SLURM nodes to serve a larger model.
Generate the SLURM submission script without submitting the job, to check it first.
Send a test chat request to a running server to confirm it responds correctly.
Hot-reload updated model weights into a running server during an RL training run.

This is a saved copy of the skill's SKILL.md instructions. It may differ from the upstream file or the version you install.

In this document (6)

Inference Server

Starting the server

Always use the inference entry point — never vllm serve or python -m vllm.entrypoints.openai.api_server directly. The entry point runs setup_vllm_env() which configures environment variables (LoRA, multiprocessing) before vLLM is imported.

# With a TOML config
uv run inference @ path/to/config.toml

# With CLI overrides
uv run inference --model.name Qwen/Qwen3-0.6B --model.max_model_len 2048 --model.enforce_eager

# Combined
uv run inference @ path/to/config.toml --server.port 8001 --gpu-memory-utilization 0.5

SLURM scheduling

The inference entrypoint supports optional SLURM scheduling, following the same patterns as SFT and RL.

Single-node SLURM

# inference_slurm.toml
output_dir = "/shared/outputs/my-inference"

[model]
name = "Qwen/Qwen3-8B"

[parallel]
tp = 8

[slurm]
job_name = "my-inference"
partition = "cluster"

uv run inference @ inference_slurm.toml

Multi-node SLURM (independent vLLM replicas)

Each node runs an independent vLLM instance. No cross-node parallelism — TP and DP must fit within a single node's GPUs.

# inference_multinode.toml
output_dir = "/shared/outputs/my-inference"

[model]
name = "PrimeIntellect/INTELLECT-3-RL-600"

[parallel]
tp = 8
dp = 1

[deployment]
type = "multi_node"
num_nodes = 4
gpus_per_node = 8

[slurm]
job_name = "my-inference"
partition = "cluster"

Dry run

Add dry_run = true to generate the sbatch script without submitting:

uv run inference @ config.toml --dry-run true

Custom endpoints

The server extends vLLM with:

/v1/chat/completions/tokens — accepts token IDs as prompt input (used by multi-turn RL rollouts)
/update_weights — hot-reload model weights from the trainer
/load_lora_adapter — load LoRA adapters at runtime
/init_broadcaster — initialize weight broadcast for distributed training

Testing the server

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-0.6B",
    "messages": [{"role": "user", "content": "Hi"}],
    "max_tokens": 50
  }'

Key files

src/prime_rl/entrypoints/inference.py — entrypoint with local/SLURM routing
src/prime_rl/inference/server.py — vLLM env setup
src/prime_rl/configs/inference.py — InferenceConfig and all sub-configs
src/prime_rl/inference/vllm/server.py — FastAPI routes and vLLM monkey-patches
src/prime_rl/templates/inference.sbatch.j2 — SLURM template (handles both single and multi-node)
configs/debug/infer.toml — minimal debug config

Local tool required

1 FOR YOU

01
Install prime-rl and its dependencies
Run uv sync (or uv sync --all-extras to include optional extras such as flash-attn) so the inference entry point and its dependencies are available before starting the server.
```
uv sync
```
```
uv sync --all-extras
```

Listed as free: Connected services may require an account or charge fees.

Not standalone. This skill ships with huggingface/prime-rl and only works together with that tool — install the tool first, then add this skill.

Run this outside your AI tool, using Node.js 22.20.0 or newer. Skills CLI 1.7.2 skips skill selection; prompts cancel installation without an interactive terminal. If choosing Project when asked about installation scope, open your project's folder before running this.

Install checklist

Check selected tools if a picker appears; Claude Code may be unchecked under Additional agents. ↑/↓ moves, Space toggles selections, Enter confirms.
Without a tool picker, check Installing to: for your intended tool. Stop if it names a different tool.
If asked, choose Symlink (Recommended) for shared files or Copy to all agents for separate copies; review the summary before confirming installation.
Install the find-skills skill? defaults to Yes, adding another skill for all projects. Choose No if you don't want that extra installation.

Skills CLI 1.7.2 sends install data to skills.sh by default, including the source, skill names, target tools and file details; GitHub sources are reported only when GitHub confirms the repository is public, while other remote sources can also be reported. Confirmed-public GitHub sources also trigger a security-audit request. Before installing, set the environment variable DO_NOT_TRACK=1 or DISABLE_TELEMETRY=1 to disable both.

npx skills add huggingface/prime-rl --skill "inference-server" --full-depth

After installing for Claude Code, open it in the installation project if you chose Project; Global makes the skill available across projects on this computer. Once Claude Code loads the skill, its description helps Claude decide when to use it.

If installation created a top-level skills folder that did not exist when your Claude Code session started, run /reload-skills in that session. Run it again after later changes in that folder.

What's inside

Files included with this skill.