Summary
Automation for software that has no API.
Everything here happens through the screen, so it reaches applications no integration covers — and for the same reason it acts on whatever is in front of it, which is why the window-name setting matters more than it looks.
What it is
A local MCP server that analyzes the screen with Microsoft's OmniParser to find the interactive elements, then operates the GUI directly. The author confirms it on Windows.
What you get
- Screen analysis and GUI operation across the whole display, or scoped to one window by setting `TARGET_WINDOW_NAME`
- OmniParser inference moved off the machine you are working on — point `OMNI_PARSER_SERVER` at an address such as `127.0.0.1:8000` and start that side with `uv run omniparserserver`
- SSE instead of stdio by setting `SSE_HOST` and `SSE_PORT`
- A compatibility switch for clients other than Claude Desktop: set `OMNI_PARSER_BACKEND_LOAD` to `1`
- OmniParser's own settings exposed as environment variables — `SOM_MODEL_PATH`, `CAPTION_MODEL_NAME`, `CAPTION_MODEL_PATH`, `OMNI_PARSER_DEVICE`, `BOX_TRESHOLD`
Requirements
Clone with `--recursive`, run `uv sync`, set `OCR_LANG`, then `uv run download_models.py` to fetch the models before first use. Set `PYTHONIOENCODING` and `OCR_LANG` in the client config. Licensing is split: the project is MIT, the OmniParser repository is CC-BY-4.0, and each OmniParser model carries its own license.