Name: Omniparser Autogui MCP
Author: NON906

LABSCO SUMMARY

Automation for software that has no API.

Everything here happens through the screen, so it reaches applications no integration covers — and for the same reason it acts on whatever is in front of it, which is why the window-name setting matters more than it looks.

What it is

A local MCP server that analyzes the screen with Microsoft's OmniParser to find the interactive elements, then operates the GUI directly. The author confirms it on Windows.

What you get

Screen analysis and GUI operation across the whole display, or scoped to one window by setting TARGET_WINDOW_NAME
OmniParser inference moved off the machine you are working on — point OMNI_PARSER_SERVER at an address such as 127.0.0.1:8000 and start that side with uv run omniparserserver
SSE instead of stdio by setting SSE_HOST and SSE_PORT
A compatibility switch for clients other than Claude Desktop: set OMNI_PARSER_BACKEND_LOAD to 1
OmniParser's own settings exposed as environment variables — SOM_MODEL_PATH, CAPTION_MODEL_NAME, CAPTION_MODEL_PATH, OMNI_PARSER_DEVICE, BOX_TRESHOLD

Requirements

Clone with --recursive, run uv sync, set OCR_LANG to the OCR language — optional in the code, but the README sets it to en, with export instead of set outside Windows — then uv run download_models.py to fetch the models before first use. Set PYTHONIOENCODING, and the same OCR_LANG, in the client config. Licensing is split: the project is MIT, the OmniParser repository is CC-BY-4.0, and each OmniParser model carries its own license.

Setup effort

Build from source — git clone --recursive https://github.com/NON906/omniparser-autogui-mcp.git

See installation options

USE CASES

01

Driving a desktop application that offers no integration

The screen is parsed into elements the model can name, and it clicks and types against them.

02

Keeping the agent inside one window

Name the window and the server operates on that alone rather than the whole desktop.

03

Running the vision model somewhere with a GPU

Start the parser as its own server on the stronger machine and point the MCP side at its address.

04

Searching in the browser that is already open

The author's own example: ask for a search in the on-screen browser, and it types into the window rather than opening a new session.

BEFORE YOU INSTALL

What to know first

Requires

uv on PATH
Git, for the clone with --recursive
Windows — the only platform the author confirms

Optional

OCR_LANG in your environment

Where it runs

The documented client runs the server on your machine.

INSTALLATION

Connect your client

Server setup

TERMINAL

git clone --recursive https://github.com/NON906/omniparser-autogui-mcp.git
cd omniparser-autogui-mcp
uv sync
set OCR_LANG=en    # Windows (cmd); on macOS/Linux: export OCR_LANG=en
uv run download_models.py

MCP Clients

Claude Desktop

CLAUDE DESKTOP · JSON

{
  "mcpServers": {
    "omniparser_autogui_mcp": {
      "command": "uv",
      "args": [
        "--directory",
        "D:\\CLONED_PATH\\omniparser-autogui-mcp",
        "run",
        "omniparser-autogui-mcp"
      ],
      "env": {
        "PYTHONIOENCODING": "utf-8",
        "OCR_LANG": "en"
      }
    }
  }
}

Omniparser Autogui MCP