Summary
A real browser an agent can work in.
Rendering, interaction and the console log are all on the table: a page that only assembles itself in a browser can be scrolled to the bottom, filled in, screenshotted and read - and when none of the shaped calls fit, there is always running JavaScript in the page.
What it is
Browser automation over Puppeteer, with Steel managing the browser sessions. The page is a real rendered browser rather than a fetched document, so what the tools see is what a person would see.
What you get
- Navigation to any URL with the wait condition and timeout under your control - load, DOM content, or network idle
- Screenshots of the whole page or of a single element by selector, kept as named resources you can fetch again
- Clicking, hovering, filling inputs and choosing from dropdowns, all by CSS selector
- Page content extracted whole, or only from the elements a selector matches
- Scrolling to trigger lazy-loading, with the delay between scrolls and the maximum number of scrolls both settable
- Arbitrary JavaScript executed in the page, for whatever the shaped calls do not cover
- Browser console output available as a resource, so what the page logged is readable after the fact
Requirements
Node, and a Steel session for the browser. Everything is addressed by CSS selector, so a site that changes its markup will need those selectors revisited.
Setup effort
Build from source — clone the repository and build it, then point your client at the binary