Labsco
ARAS-Workspace logo

Claude KVM

โ˜… 15

from ARAS-Workspace

๐Ÿค– โšก๏ธ MCP server (๏ฃฟ MacOS) โ€” control remote desktops via VNC

๐Ÿ”ฅ๐Ÿ”ฅ๐Ÿ”ฅโœ“ VerifiedAccount requiredAdvanced setup

Claude KVM

Claude KVM

Remote Access, Artificial Intelligence

claude-kvm.ai

Claude KVM is an MCP tool that controls remote desktop environments over VNC. It consists of a thin JS proxy layer (MCP server) and a platform-native Swift VNC daemon running on your macOS system.

Claude KVM Demo Claude KVM Demo Mac

[!TIP] Phantom-WG could be a great alternative for you. Isolate your VNC server within your own network while enjoying self-hosted VPN performance with the extra privacy features you gain along the way.

Live Test Runs

[!NOTE] Tests are conducted transparently on GitHub Actions โ€” each step is visible in the CI environment. At the end of every test, whether the integration passes or fails, you'll find screenshots of each step the agent took during the session, along with an .mp4 video recording that captures the entire session. By reviewing these recordings and screenshots, you can observe how the agent progressed through each stage, how long the task took, and what decisions were made based on the system prompt. You can use these examples as a reference when crafting your own system prompts or instructions for the MCP server in your own environment.

[!WARNING] Artifacts attached to these runs may have expired due to GitHub's artifact retention policy. Persistent copies are prepared via the Persist Artifacts workflow and can always be accessed by run ID from the artifacts/ directory on the press-kit branch.

Architecture

graph TB
    subgraph MCP["MCP Client (Claude)"]
        AI["Claude"]
    end

    subgraph Proxy["claude-kvm ยท MCP Proxy (stdio)"]
        direction TB
        Server["MCP Server<br/><code>index.js</code>"]
        Tools["Tool Definitions<br/><code>tools/index.js</code>"]
        Server --> Tools
    end

    subgraph Daemon["claude-kvm-daemon ยท Native VNC Client (stdin/stdout)"]
        direction TB
        CMD["Command Handler<br/><i>PC Dispatch</i>"]
        Scale["Display Scaling<br/><i>Scaled โ†” Native</i>"]

        subgraph Screen["Screen"]
            Capture["Frame Capture<br/><i>PNG ยท Crop ยท Diff</i>"]
            OCR["OCR Detection<br/><i>Apple Vision</i>"]
        end

        subgraph InputGroup["Input"]
            Mouse["Mouse<br/><i>Click ยท Drag ยท Move ยท Scroll</i>"]
            KB["Keyboard<br/><i>Tap ยท Combo ยท Type ยท Paste</i>"]
        end

        VNC["VNC Bridge<br/><i>LibVNCClient 0.9.15</i>"]

        CMD --> Scale
        Scale --> Capture
        Scale --> Mouse
        Scale --> KB
        Capture -.->|"framebuffer"| VNC
        Mouse -->|"pointer events"| VNC
        KB -->|"key events"| VNC
    end

    subgraph Target["Target Machine"]
        VNC_Server["VNC Server<br/><i>:5900</i>"]
        Desktop["Desktop Environment"]
        VNC_Server --> Desktop
    end

    AI <-->|"stdio<br/>JSON-RPC"| Server
    Server <-->|"stdin/stdout<br/>PC (NDJSON)"| CMD
    VNC <-->|"RFB Protocol<br/>TCP :5900"| VNC_Server

    classDef proxy fill:#1a1a2e,stroke:#16213e,color:#e5e5e5
    classDef daemon fill:#0f3460,stroke:#533483,color:#e5e5e5
    classDef target fill:#1a1a2e,stroke:#e94560,color:#e5e5e5

    class Server,Tools proxy
    class CMD,Scale,VNC,Capture,Mouse,KB daemon
    class VNC_Server,Desktop target

Layers

LayerLanguageRoleCommunication
MCP ProxyJavaScript (Node.js)Communicates with Claude over MCP protocol, manages daemon lifecyclestdio JSON-RPC
VNC DaemonSwift/C (Apple Silicon)VNC connection, screen capture, mouse/keyboard input injectionstdin/stdout PC (NDJSON)

PC (Procedure Call) Protocol

Communication between the proxy and daemon uses the PC protocol over NDJSON:

Request:      {"method":"<name>","params":{...},"id":<int|string>}
Response:     {"result":{...},"id":<int|string>}
Error:        {"error":{"code":<int>,"message":"..."},"id":<int|string>}
Notification: {"method":"<name>","params":{...}}

Coordinate Scaling

The VNC server's native resolution is scaled down to fit within --max-dimension (default: 1280px). Claude works more consistently with scaled coordinates โ€” the daemon handles the conversion in the background:

Native:  4220 x 2568  (VNC server framebuffer)
Scaled:  1280 x 779   (what Claude sees and targets)

mouse_click(640, 400) โ†’ VNC receives (2110, 1284)

Screen Strategy

Claude minimizes token cost with a progressive verification approach:

diff_check       โ†’  changeDetected: true/false     ~5ms    (text only, no image)
detect_elements  โ†’  OCR text + bounding boxes      ~50ms   (text only, no image)
cursor_crop      โ†’  crop around cursor              ~50ms   (small image)
screenshot       โ†’  full screen capture             ~200ms  (full image)

detect_elements uses Apple Vision framework for on-device OCR. Returns text content with bounding box coordinates in scaled space โ€” enables precise click targeting without consuming vision tokens.


Tools

All operations are performed through a single vnc_command tool:

Screen

ActionParametersDescription
screenshotFull screen PNG capture
cursor_cropCrop around cursor with crosshair overlay
diff_checkDetect screen changes against baseline
set_baselineSave current screen as diff reference

Mouse

ActionParametersDescription
mouse_clickx, y, button?Click (left|right|middle)
mouse_double_clickx, yDouble click
mouse_movex, yMove cursor
hoverx, yMove + settle wait
nudgedx, dyRelative cursor movement
mouse_dragx, y, toX, toYDrag from start to end
scrollx, y, direction, amount?Scroll (up|down|left|right)

Keyboard

ActionParametersDescription
key_tapkeySingle key press (enter|escape|tab|space|...)
key_combokey or keysModifier combo ("cmd+c" or ["cmd","shift","3"])
key_typetextType text character by character
pastetextPaste text via clipboard

Detection

ActionParametersDescription
detect_elementsOCR text detection with bounding boxes (Apple Vision)

Returns text elements with bounding box coordinates in scaled space:

{"method":"detect_elements"}
{"result":{"detail":"13 elements","elements":[{"confidence":1,"h":9,"text":"Finder","w":32,"x":37,"y":6},{"confidence":1,"h":9,"text":"File","w":15,"x":84,"y":6},{"confidence":1,"h":9,"text":"Edit","w":19,"x":112,"y":6},{"confidence":1,"h":9,"text":"View","w":22,"x":143,"y":6},{"confidence":1,"h":11,"text":"Go","w":15,"x":179,"y":6},{"confidence":1,"h":9,"text":"Window","w":35,"x":207,"y":6},{"confidence":1,"h":11,"text":"Help","w":22,"x":255,"y":6},{"confidence":1,"h":11,"text":"8โ€ข","w":26,"x":1161,"y":6},{"confidence":1,"h":9,"text":"Fri Feb 20 22:19","w":80,"x":1189,"y":6},{"confidence":1,"h":9,"text":"Assets","w":32,"x":1202,"y":97},{"confidence":1,"h":9,"text":"Passwords.kdbx","w":74,"x":1181,"y":168},{"confidence":1,"h":93,"text":"PHANTOM","w":633,"x":322,"y":477},{"confidence":1,"h":32,"text":"YOUR SERVER, YOUR NETWORK, YOUR PRIVACY","w":629,"x":325,"y":568}],"scaledHeight":717,"scaledWidth":1280}}

Configuration

ActionParametersDescription
configure{<params>}Set timing/display params at runtime
configure{reset: true}Reset all params to defaults
get_timingGet current timing + display params

Control

ActionParametersDescription
waitms?Wait (default 500ms)
healthConnection status + display info
shutdownGraceful daemon shutdown

Authentication

Supported VNC authentication methods:

  • VNC Auth โ€” password-based challenge-response (DES)
  • ARD โ€” Apple Remote Desktop (Diffie-Hellman + AES-128-ECB)

macOS is auto-detected via the ARD auth type 30 credential request. When detected, Meta keys are remapped to Super (Command key compatibility).


MCP Badge

[!NOTE] Running on a bare-metal Mac? See the Mac M1 Preparation Tricks for VNC hardening, SSH tunneling, and session stability tips.


"Claude" is a trademark of Anthropic, PBC. This project is not affiliated with or endorsed by Anthropic.

Copyright (c) 2026 Riza Emre ARAS โ€” MIT License