A screenshot tells the model what the screen looks like; the UI layout call tells it where to tap. Having both is what turns "describe this screen" into "press that button".
A thin server over the Android Debug Bridge. It picks the connected device automatically when there is only one, and otherwise takes the device serial from a config file.
- `execute_adb_command` — run any ADB command and get its output back, which is the escape hatch that covers everything else
- `get_screenshot` — a screenshot of the device, returned as an image
- `get_uilayout` — the clickable elements on the current screen with their text, content description, bounds and centre coordinates, which is what makes tapping the right thing possible
- `get_packages` — every package installed on the device
- `get_package_action_intents` — the non-data actions from the Activity Resolver Table for a package
Python and ADB installed and configured, plus an Android device or emulator connected. The project uses `uv`: `uv python install 3.11` then `uv sync`. Configuration is optional — with exactly one device connected it connects automatically. With several connected you must create `config.yaml` from the example and set `device.name` to a serial from `adb devices`; without it the server reports an error listing the available devices rather than picking one. Register it in the client by pointing `command` at `uv` with `--directory` set to your clone and `run server.py` as the arguments.
Build from source — clone the repository and build it, then point your client at the binary