ScreenHand

Let AI control your desktop — click buttons, fill forms, automate workflows in ~50ms with zero extra AI calls.

An open-source MCP server for macOS and Windows. Works with Claude, Cursor, Codex CLI, and any MCP-compatible client.

Quick Start | What It Does | Example | All 111 Tools | Architecture | Website

The Problem

AI assistants can write code but can't use your computer. Every click requires a screenshot → LLM interpretation → coordinate guess — 3-5 seconds and an API call per action.

ScreenHand gives AI direct access to native OS APIs. No screenshots needed for clicks. No AI calls for button presses.

Prebuilt Platform Knowledge

Every install ships with battle-tested knowledge so AI starts from EXPERT level on day one — no re-exploration needed:

These load automatically when the matching app or website is detected. No setup required.

Verify after install:

npx screenhand --info

What It Does

ScreenHand gives AI agents eight capabilities:

Desktop Control — 19 tools

Click buttons, type text, read UI trees, navigate menus, drag, scroll — all via native Accessibility APIs in ~50ms. Works with any app: Finder, Notes, VS Code, Xcode, System Settings, etc.

Browser Automation — 15 tools

Full Chrome control via DevTools Protocol. Navigate, click, type, run JavaScript, fill forms — all in the background at ~10ms. Built-in anti-detection (browser_stealth, browser_human_click) for sites with bot protection.

Smart Fallbacks — 8 tools

click_with_fallback, type_with_fallback, etc. automatically try Accessibility → CDP → OCR → coordinates. You don't have to pick the right method — ScreenHand figures it out.

Memory & Learning — 14 tools

Gets smarter every session. Logs tool calls, saves winning strategies, tracks error patterns with fixes. Zero config, zero latency overhead (in-memory cache, async disk writes). Ships with 12 seed strategies for common macOS workflows. 6 learning policies: locator stability, sensor effectiveness, recovery ranking, pattern recognition, adaptive timing, and topology (navigation edge reliability).

App Mastery Map — automatic per-app spatial understanding

Builds a persistent reverse-engineered blueprint of every app from normal tool usage. 8 features record automatically: page zones, navigation graph (BFS pathfinding), hierarchy, I/O contracts, state machine, element visibility, timing profiles, and ready signals. Mastery levels (beginner → pro → expert → grandmaster) honestly reflect how well ScreenHand knows each app. Maps stored at ~/.screenhand/app-maps/.

Website Feature Discovery — real features, not generic ladders

discover_features fetches an app's official website and extracts real product features (headings, feature cards, definition lists). Assigns difficulty tiers automatically and generates value-add features only ScreenHand can provide: bulk operations, cross-app export, content summarization, auto-organize, and change monitoring. No LLM calls needed — pure rule-based extraction. Features merge into the reference file and enrich the mastery ladder.

Jobs & Orchestration — 34 tools

Queue multi-step jobs, run them via background worker daemon, coordinate multiple AI agents with session leases, detect stalls, auto-recover. Survives client restarts.

Perception & Planning — 17 tools

Continuous screen awareness (3-rate perception loop at 100ms/300ms/1000ms), real-time world model with entity tracking, goal-oriented planning with auto-decomposition, recovery engine with self-healing. The system always knows what's on screen and feeds observations into the App Mastery Map.

Full reference: See all 111 tools with descriptions.

Example

Browser — Claude controls Chrome in the background while you work:

You: Search for "screenhand" on Instagram

→ browser_tabs()                                        # ~10ms
  [34DF5DE1] Instagram — https://www.instagram.com/

→ browser_js({ code: "/* click Search icon */" })       # ~10ms
→ browser_fill_form({ selector: "input", text: "screenhand" })  # ~50ms (human-like)
→ browser_js({ code: "/* extract results */" })         # ~10ms

Found @screenhand_ as the top result.

Desktop — native app control without screenshots:

→ apps()                     # List running apps           ~10ms
→ focus("com.apple.Notes")   # Bring Notes to front        ~10ms
→ ui_tree()                  # Read full UI element tree    ~50ms
→ ui_press("New Note")       # Click "New Note" button     ~50ms
→ type_text("Hello world")   # Type text                   ~30ms

Cross-app — chain actions across your whole desktop:

→ browser_js(...)            # Extract data from Chrome
→ focus("com.apple.Notes")   # Switch to Notes
→ type_text(extractedData)   # Paste it in
→ key("cmd+s")               # Save

Claude Code Plugin

If you use Claude Code, ScreenHand includes a plugin with 13 skills and 5 agents that wrap all 111 tools into intent-oriented workflows.

./install-plugin.sh   # after npm install && npm run build:native

5 specialized agents: marketing, design, QA, scraper, orchestrator.

How It Works

AI Client (Claude, Cursor, Codex CLI)
    ↓ MCP protocol (stdio)
ScreenHand MCP Server (TypeScript)
    ↓ JSON-RPC (stdio)
Native Bridge (Swift on macOS / C# on Windows)
    ↓ OS APIs
Accessibility, CoreGraphics, Vision, UI Automation, SendInput

ScreenHand reads the UI tree and DOM directly — no screenshots needed for most operations. When screenshots are needed (canvas apps, visual verification), OCR runs in ~600ms via the native Vision framework.

Docs

Contributing

git clone https://github.com/manushi4/screenhand.git
cd screenhand && npm install && npm run build:native
npm test   # 1331 tests, 54 files

Contact

• Email: khushi@clazro.com
• Issues: github.com/manushi4/screenhand/issues
• Website: screenhand.com

License

AGPL-3.0-only — Copyright (C) 2025-2026 Clazro Technology Private Limited

screenhand.com | khushi@clazro.com | A product of Clazro Technology Private Limited

Quick Start

1. Add to your AI client (one step)

claude mcp add screenhand -- npx -y screenhand

Done. That's it.

Add to ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "screenhand": {
      "command": "npx",
      "args": ["-y", "screenhand"]
    }
  }
}

Add to .cursor/mcp.json:

{
  "mcpServers": {
    "screenhand": {
      "command": "npx",
      "args": ["-y", "screenhand"]
    }
  }
}

Add to ~/.codex/config.toml:

[mcp.screenhand]
command = "npx"
args = ["-y", "screenhand"]
transport = "stdio"

ScreenHand is a standard MCP server over stdio. Run with npx -y screenhand.

2. Grant permissions

macOS: System Settings > Privacy & Security > Accessibility > enable your terminal app.

Windows: No special permissions needed.

3. Browser control (optional)

Launch Chrome with remote debugging to enable browser tools:

open -a "Google Chrome" --args --remote-debugging-port=9222

That's it. Your AI client now has 111 tools for desktop automation — and ships with prebuilt knowledge for 36 apps so you don't start from zero.

git clone https://github.com/manushi4/screenhand.git
cd screenhand && npm install && npm run build:native

On Windows, use npm run build:native:windows instead.

Requirements

FAQ

Computer Use is cloud-based and screenshot-driven. ScreenHand is local-first, uses native OS APIs (50ms vs 3-5s per action), costs zero API calls for clicks/typing, and runs entirely on your machine.

Any app with Accessibility support (most macOS/Windows apps). Chrome and Electron apps get full DOM access via CDP. Canvas-heavy apps (games, Photoshop viewport) use OCR as fallback.

Ships with EXPERT-level prebuilt knowledge for: Terminal, Mail, Finder, Calendar, Reminders, Keynote, Pages, Notes, Photos, Apple Music, WhatsApp, Figma, Discord, DaVinci Resolve, Canva, Instagram, X/Twitter, LinkedIn, YouTube, Reddit, Notion, n8n, and more. Any other app gets explored and learned automatically on first use.

Runs locally, never sends screen data externally. PII is redacted from all persisted data (memory, playbooks, strategies). Dangerous protocols (javascript:, data:) are blocked. AppleScript and browser JS execution are audit-logged.

Yes. Session leases with heartbeat prevent conflicts. The supervisor daemon detects stalls and recovers. Each agent claims its own app window.

Accessibility: ~50ms. Chrome CDP: ~10ms (background, no focus needed). OCR: ~600ms. Memory lookups: ~0ms (in-memory cache). All disk writes are async and non-blocking.