A provider dashboard tells you whether an API is up; this tells you what your own calls are doing right now, against thresholds you chose. Because it is the same tool in a terminal, in CI and over MCP, an agent can confirm the model it is about to depend on is healthy before committing to the work.
A Go CLI for LLM serving reliability, with an MCP server built in. It sends a streaming request to each configured provider and model, then reports what users actually feel: how long before the first token arrives, how long the whole response takes, and whether the endpoint counts as healthy, degraded or erroring against thresholds you set.
- Probe every provider and model in `probes.yml` at once and read back TTFT, latency, throughput and health per model — `probe_all`
- Probe one model with no config file at all, given a provider, a model name and the env var holding the key — `probe_model`
- List the providers and models already configured, with their thresholds, before deciding what to probe — `list_providers`
- Return the full parsed configuration including defaults, providers, models and thresholds — `get_config`
- OpenAI, Anthropic, Google, Azure OpenAI and AWS Bedrock are handled natively; anything OpenAI-compatible works by setting `base_url`
The `llmprobe` binary — a prebuilt release for Linux, macOS or Windows, or `go install github.com/Jwrede/llmprobe@latest`. Register it with `claude mcp add --transport stdio llmprobe -- llmprobe mcp`, or install the Claude Code plugin with `claude plugin install Jwrede/llmprobe`. The config-reading tools want a `probes.yml` holding providers, models and thresholds; API keys go in as `${ENV_VAR}` references, so the file itself carries no secrets. It contacts only the endpoints you configure.
One command plus a key — go install github.com/Jwrede/llmprobe@latest, then supply credentials