Everything runs over SSH into a guest you built, so a wrong click or a bad shell command hits the VM rather than your host. What makes it workable is the targeting: text search, magnify-and-map, and numbered marks each remove a place where a model would otherwise guess at coordinates. One rule to know up front — a screenshot requires an active project.
A Python server that drives a Linux VM over SSH: X11 mouse and keyboard control, screenshots, OCR-based clicking, and shell commands, all confined to the guest.
- Input control — `move_mouse`, `click`, `click_in_window`, `drag`, `scroll`, `type_text`, `press_keys`, `key_down` and `key_up`, `set_clipboard`, `paste`
- Clicking by what's on screen rather than by pixel — `find_text` OCRs the screen and returns every match with coordinates, `click_text` clicks the match's exact centre
- `zoom` magnifies a region and keeps the crop mapping, so `click_zoomed` turns a point in the zoomed image back into the right full-screen pixel
- `mark_screen` numbers every detected text element and `click_mark` picks one by number — a discrete choice instead of an estimate
- `run_actions` executes a whole sequence of UI actions in a single call
- Shell and file transfer — `ssh_execute`, `ssh_upload`, `ssh_download`, `ssh_connection_info`; screenshots via `take_screenshot`, stored in the project and exposed as resources
- Session memory — `project_init`, `project_log`, `project_save_result` and `project_save_advice`, so one run leaves notes for the next
Host: Python 3.12+, uv or pip, and QEMU/KVM with libvirt. Guest: Linux with an X11 desktop, an SSH server, and `openssh`, `xdotool`, `scrot`, `xrandr` and `xinput` installed. Connection settings come from `VM_HOST`, `VM_USER`, `VM_PORT`, `VM_DISPLAY` and optionally `VM_IDENTITY`. The OCR and zoom tools additionally need `tesseract` on the host plus pillow and pytesseract — the rest of the server runs without them.