VisionAX

A short one, on what happens when the accessibility tree goes dark.

Everything I've written about Mary so far rests on one commitment: the accessibility tree is ground truth. AX gives you structure an app has already agreed to expose — roles, frames, a walkable hierarchy — without ever taking a screenshot of the user's screen. That's the whole privacy posture in one sentence, and it's the reason computer use can be legible instead of a black box guessing at pixels.

The commitment holds until an app declines to participate. A canvas-drawn Electron surface, a DRM'd video element, a custom-rendered control stack — AX reports a single opaque node and nothing underneath it. At that point the honest choices are fail, or fall back to vision. And a vision fallback that phones out to a cloud model or a screenshot-scraping service would quietly break the exact promise the rest of the architecture makes. So the fallback has to earn the same trust AX already has, not spend it.

That's what VisionAX is: an isolated, GPL-3.0 OpenCV engine for macOS — a C++ core behind a C header, a Swift face on top, and a bench app to watch what it sees. It ships as its own SwiftPM package rather than living inside Mary, and Mary depends on it the ordinary way a library gets consumed, not the way a vendored fallback gets smuggled in.

The same shape as AX, on purpose

The one design decision that matters more than the CV pipeline underneath it: VisionAX doesn't invent a new tree format for detected UI. It publishes regions in exactly the shape of Mary's own AXNodeSnapshot / AXWindowSnapshot value tree. A detected box and a walked accessibility node are interchangeable to anything downstream — same JSON shape, same traversal code, same consumer. Vision degrades gracefully into the interface AX already defined, instead of becoming a second API surface that computer-use logic has to branch on.

The pipeline itself is unglamorous by design: Canny edge detection over a screenshot, contour extraction, a filter-and-dedupe pass to collapse a stroked rectangle's inner and outer edges into one box, then a tree builder that nests each region under the smallest kept box that contains it — the same containment logic AX uses to nest a control inside a group inside a window.

That's the bench, mid-run, over an ordinary web page. Every orange box is a VXRegion — the tree on the left is the same AXWindow / node hierarchy Mary already knows how to walk, just populated by pixels instead of an app's own AX implementation. 245 nodes out of 945 raw contours, five milliseconds, running entirely on-device.

No labels yet, and that's deliberate

Every detected node ships with role VXRegion, category .other, and label: nil. No OCR, no classifier, no guess at what a box is — only that a boundary exists there. That's a narrower claim than a real AX node makes, and I'd rather ship the narrower true claim than a placeholder label that reads as confident and is indistinguishable from a correct one until something breaks. Labeling is a real step, and it comes later, as its own seam.

The commitment this preserves

Nothing about this fallback changes the trust boundary. The input is a screenshot the machine already has; the output is bounding-box math computed by an inspectable engine running in-process. No network call, no third-party model, no image ever leaves the device to get a tree back. When AX is available, Mary uses AX. When it isn't, the fallback is vision that behaves like AX — legible, local, and built to the same shape — rather than vision that behaves like surveillance. Computer use should degrade in capability before it ever degrades in privacy.