Labsco
microsoft logo

winapp-ui-automation

✓ Official★ 1,128

by microsoft · part of microsoft/winappcli

Inspect and interact with running Windows app UIs from the command line using UI Automation (UIA). Use when an AI agent or developer needs to inspect a UI element tree, find controls, take screenshots, click buttons, read or set text, or verify UI state in a running Windows app. Works with any framework WinUI 3, WPF, WinForms, Win32, Electron.

🧰 Not standalone. This skill ships with microsoft/winappcli and only works together with that tool — install the tool first, then add this skill.

This is the playbook your agent receives when the skill activates — you don't need to read it to use the skill, but it's here to audit before installing.

When to use

  • Inspecting a running Windows app's UI from the command line
  • AI agents interacting with Windows applications (clicking buttons, reading text, taking screenshots)
  • Verifying UI state during development or testing
  • Automating UI workflows without Playwright or Selenium
  • Debugging WinUI 3, WPF, WinForms, Win32, or Electron app UIs

Coordinating with other UI workflows

Windows has one foreground window, one keyboard focus, one cursor, and one input stream. winapp ui therefore makes desktop-driving commands take cooperative turns so concurrent workflows cannot steal each other's focus or dismiss each other's menus. That is always on. Read-only commands never wait.

Keeping the desktop across commands is opt-in — without an id, each command is a one-shot that releases the desktop the moment it finishes:

# Set once per logical UI workflow — same value for cooperating calls, different values for
# independent workflows (even from the same agent).
$env:WINAPP_UI_WORKFLOW_ID = [guid]::NewGuid().ToString()

Rules that matter when driving this from an agent:

  • Each tool call usually gets a fresh shell, and there is no process-ancestry fallback, so commands are grouped ONLY by the id you inject. Without it every call is its own workflow.
  • A workflow with an id keeps its turn for four seconds after its last command. That covers back-to-back commands in one script; it deliberately expires while you are reasoning. Treat it as a fallback for when you cannot say you are done — not as the way to finish.
  • Run winapp ui yield when you finish a known sequence. It hands the desktop over immediately instead of making a waiting workflow sit out a four-second grace nobody needs. Yielding twice, or after the grace lapsed, is a harmless success.
  • After a reasoning gap, replay your setup. Another workflow may have used the desktop, so reopen the menu / re-navigate, re-resolve the element, then act. Do not assume transient UI survived.
  • Prefer one tight script over many round trips for a known sequence: winapp ui invoke View -w $hwnd; winapp ui search "Status bar" -w $hwnd; winapp ui click "Status bar" -w $hwnd; winapp ui yield.
  • record shares the turn with its own workflow, so same-workflow clicks and typing are captured while it runs — but only if both commands carry the same id. A record with no id blocks everyone else for its whole duration.
  • Ordering is owner affinity, then FIFO. An active workflow may keep issuing commands ahead of others already waiting; once it yields or its grace expires, waiters are served in arrival order.
  • Waiting is indefinite and cancellable. A status line appears after one second; Ctrl+C exits 130 with error code cancelled and the command never ran.
  • There is no hard cap — a long script, unbounded recording, or failure loop can block other mutating workflows until it finishes or is stopped.

Commands that never wait: status, list-windows, inspect, search, get-*, wait-for. Commands that wait for the turn but never take the desktop (headless/locked-session friendly): set-value, scroll-into-view, scroll --direction/--to, record. They mutate the app, so they queue behind another workflow. Inside your own workflow they overlap with other shared work — that is how record captures the set-value calls it is recording — but they still wait behind an earlier DesktopExclusive command of your own workflow, so a click followed by a set-value runs in the order you wrote it. Commands that take the desktop exclusively: invoke, click, drag, hover, scroll --wheel, touch, pen, focus, send-keys, screenshot.

# Finish a workflow deliberately rather than leaving the desktop reserved for four more seconds.
winapp ui yield

Common patterns

Discover and interact

# See what's clickable, then screenshot for context
winapp ui inspect -a myapp --interactive; winapp ui screenshot -a myapp

# Click and verify the page changed
winapp ui invoke btn-settings-a1b2 -a myapp; winapp ui wait-for pn-settingspage-c3d4 -a myapp --timeout 3000; winapp ui screenshot -a myapp

# Fill a form and submit
winapp ui set-value txt-searchbox-e5f6 "hello" -a myapp; winapp ui invoke btn-submit-7a90 -a myapp; winapp ui screenshot -a myapp

Find visible text and click it

# Search by text — output shows invokable ancestor
winapp ui search "Save changes" -a myapp
# Output:
#   lbl-savechanges-a1b2 "Save changes" (120,40 80x20)
#         ^ invoke via: btn-save-c3d4 "Save"

# Invoke by text — auto-walks to parent Button
winapp ui invoke 'Save changes' -a myapp

Navigate multi-page apps

# Click nav item, wait for page, inspect what's available
winapp ui invoke itm-samples-3f2c -a myapp; winapp ui wait-for pn-samplespage-b4e7 -a myapp; winapp ui inspect -a myapp --interactive

Disambiguate duplicate elements

# When text search matches multiple elements, the error shows slugs for each — pick the right one
winapp ui invoke Submit -a myapp
# → Selector matched 3 elements:
#   [0] Button "Submit Order" → btn-submitorder-a1b2
#   [1] Button "Submit" → btn-submit-c3d4
# Use the slug: winapp ui invoke btn-submit-c3d4 -a myapp

Key concepts

  • Selector brackets: inspect and search output shows selectors in [brackets] — use the bracketed value with other ui commands. Selectors are either AutomationId (stable, developer-set) or generated slug (e.g., btn-name-hash).
  • AutomationId selectors: When an element has a unique AutomationId, it becomes the selector directly (e.g., [MinimizeButton]). These survive layout changes and localization — preferred for stable targeting.
  • Slug selectors: When no unique AutomationId exists, a generated slug is used (e.g., [btn-close-a2b3]). Format: prefix-name-hash. May go stale after UI changes.
  • Plain text search: search and invoke accept plain text — search Minimize finds elements with "Minimize" in their Name or AutomationId (substring, case-insensitive). No special syntax needed.
  • --interactive flag: Filters to invokable elements only with auto-depth 8 — the fastest way to see what you can click
  • Invokable ancestor surfacing: When a search result isn't invokable, the nearest invokable parent is shown with its selector
  • ; chaining: Chain commands with ; to run multiple operations in one call, reducing agent round-trips
  • -a vs -w: Use -a to find apps by name/title/PID. Use -w <HWND> for stable window targeting
  • Element markers: [on]/[off] for toggles, [collapsed]/[expanded], [scroll:v]/[scroll:h]/[scroll:vh] for scrollable containers, [offscreen], [disabled], value="..." for editable elements

Tips

  • Use --interactive with inspect as your first command — it shows only what you can click
  • Chain commands with ; to reduce round-trips (see note below on why not &&)
  • Use slugs from output to target specific elements — they're hash-validated and shell-safe
  • Use plain text search to find elements: search Minimize, invoke Submit
  • When multiple elements match text search, the error shows slugs for each — pick the right one
  • Use get-property --property ToggleState to verify checkbox/toggle state after invoke
  • scroll auto-finds the nearest scrollable parent
  • Use --capture-screen to capture popup overlays, dropdown menus, and flyouts (also brings the window to the foreground)
  • Use hover before screenshot --capture-screen to capture tooltips and hover-triggered UI
  • Use --focus to foreground the target window before capture without switching to screen-DC capture (default capture path uses Windows.Graphics.Capture and works while occluded)
  • Use --hide-disabled and --hide-offscreen to reduce noise

Why ; instead of &&

Use ; (not &&) to chain commands. PowerShell's && operator can freeze when a native CLI writes to stderr or uses ANSI escape sequences — this causes a pipeline deadlock. ; runs each command unconditionally and avoids this issue. This is also better for agent workflows: you usually want the screenshot to run even if the invoke had a non-zero exit (to see what went wrong).

File dialog workaround

File open/save dialogs are standard Windows dialogs with UIA support. Interact with them using existing commands:

# 1. Trigger the dialog (e.g., click "Open File" button)
winapp ui invoke btn-openfilebtn-a2b3 -a myapp

# 2. Find the dialog window
winapp ui list-windows -a myapp
# → Shows the main window + the dialog HWND
# Note: untitled zero-size windows are hidden by default; use --show-hidden to include them

# 3. Target the dialog, type the file path, and confirm
winapp ui set-value txt-1148-c4d5 "C:\path\to\file.png" -w <dialog-hwnd>
winapp ui invoke btn-open-e6f7 -w <dialog-hwnd>

Note: The filename input in standard file dialogs typically has AutomationId 1148. Use inspect -w <dialog-hwnd> --interactive to discover the actual slugs.

JSON output envelopes (v0.3.1+)

The --json envelope for ui inspect, ui get-focused, ui search, and ui wait-for was reshaped in v0.3.1. Pre-0.3.1 parsers will silently break — most fields were renamed, removed, or moved into envelopes. Highlights:

  • ui inspect --json now nests elements under windows[].elements[] (was a flat elements[]).
  • ui get-focused --json always emits an envelope — { "hasFocus": false } or { "hasFocus": true, "element": {...} } (was bare null).
  • ui search --json / ui wait-for --json may include an invokableAncestor field (element-shaped) on each match.
  • Per-element id, parentSelector, and windowHandle are removed — use selector as the public handle.

Full schemas with examples: references/ui-json-envelope.md.

  • winapp-setup for adding Windows SDK to your project
  • winapp-package for packaging apps as MSIX

CLI reference

Run winapp <command> --help for current command options, or winapp --cli-schema for the complete machine-readable command schema.