Coordinates alone make for blind automation. Pairing gestures with a screen description means the agent can decide where to tap from what is actually rendered, and the shell call is there for everything the wrapper does not name.
A Go server that wraps ADB against one attached Android device. It covers app lifecycle, screen state, input and gestures, plus raw shell, and it can send a screenshot to a vision model such as qwen2.5-vl and return a description of the screen in words.
- Apps installed, uninstalled, launched, terminated and listed, and a check for whether a given package is there
- The screen unlocked or locked, and whether it is currently locked or awake
- Text typed and key presses sent, including long presses at a coordinate
- Swipes in all four directions
- A screenshot described in words by a vision model, when that model is enabled
- Screen size and DPI, and system information for the device
- Any adb shell command, for the things no dedicated call covers
The device's id, which adb devices prints, goes in the environment. The screen-lock passcode is optional and only needed to unlock the screen. The description feature is off by default: switch it on, then supply an API key, a base URL and a model name for the vision provider you are using. Built from source with go build.