HCI
Ambient Clients
On the macOS app I've been building — and the two companies that just shipped its silhouette.
Yesterday, Meta shipped a Mac app.
Meta AI for Mac arrived on August 19, 2026 as a 1.0 beta — Apple silicon, macOS 15 or later. It has two capabilities the phone version does not. You can attach a window to the conversation and ask about what is in it. And you can hold a shortcut, speak, and have the words land in whatever app is in front of you.
Those are two of the three things I have been building for the last ten months.
Meta and I have built the same silhouette and almost nothing else in common. The features rhyme. The architecture does not. And the architecture is the entire product.
What Meta actually shipped
Credit where it is owed: it is a well-judged app.
Quick Invoke puts a compact composer over whatever you are doing at Option-Space. Window attach, with Screen Recording and Accessibility permission, reads a window's visible text and captures a screenshot to send with your next question. Dictation types into Mail, into documents, into code editors. The sidebar carries media, artifacts, history, scheduled tasks, and a personalization surface called "About Me." You can hide the Dock icon and live entirely on a keyboard shortcut. Business accounts can connect Google Workspace, and people with professional Facebook and Instagram accounts get analytics — organic performance, comparisons against similar brands, ad campaign review, budget suggestions.
That last clause is the one worth sitting with. The reporting frames it plainly: this is an app for creators and small businesses who already live on Meta's platforms. The assistant is the doorway; the ad business is the room. That is not a scandal — it is a coherent strategy, and it is why the app exists at all. But it tells you what the app is optimizing, and it is not your privacy.
And what OpenAI shipped six days earlier
Meta's is not the only one, and it is not even the most interesting one.
On August 13, 2026, OpenAI shipped Computer History in the ChatGPT desktop app for macOS. It is opt-in, off by default, limited to Pro, Business, and Enterprise accounts, and unavailable in the EEA, Switzerland, and the UK. It replaces Chronicle, a research preview that worked by screenshotting your screen — and the replacement takes no screenshots at all. It records interaction events that macOS exposes through its accessibility system: clicks, typing, keyboard shortcuts, app switches. Those become memories and a timeline. The Register described the trade as swapping "Recall-style screenshot surveillance" for "friendly keylogging," which is unkind and not entirely wrong.
Next to it sits Computer Use: ChatGPT seeing and operating the GUI on your actual machine. It asks for Screen Recording "so ChatGPT can see the target app" and Accessibility "so ChatGPT can click, type, and navigate." Screenshots it takes are processed like any other ChatGPT content.
I am spending a section on this because OpenAI's app is the harder comparison for me to make, not the easier one. Meta's mechanism is the one I can argue against in a sentence. OpenAI arrived at the same mechanism I did — and then made three different decisions underneath it, and those decisions are the entire subject of this piece.
So the landscape is not "Meta does the invasive thing and everyone else is fine." Two companies shipped the same silhouette inside of a week, and I have been building the third one for ten months. The differences are all below the waterline.
The three questions
Any assistant that can see your screen and type into your applications has to answer three questions. Most product pages answer none of them.
- Where does the screen go?
- Where does the memory live?
- Does it act, or does it advise?
Bonnie is what happens when you answer those three first and let the feature list fall out afterwards.
Where the screen goes
When Meta's app looks at your window, it captures an image and the visible text and sends them up. That is the mechanism — it is how the feature works, not a criticism of it. The screenshot is the payload.
Bonnie does not work that way, and the reason is older than Meta's app.
When Bonnie needs to know what is in front of you, she reads the accessibility tree — the same structured description of controls and text that VoiceOver reads, requested in-process from macOS. She gets the title of the document, the text of the passage, the label on the button. She does not need the pixels, so the pixels do not move. There is a look_at_screen ability that captures a region for a vision model when the accessibility tree genuinely cannot answer — a canvas, an image, a PDF rendered as a picture — and it is deliberately the exception, gated and separate.
That separation is doctrine in the codebase, and it was paid for by a bug:
eyes are an UPGRADE for workspace apps, never a precondition for acting. Workspace focus decides which document Bonnie perceives and EDITS — never which Skills she may USE.
The failure that produced that rule is worth telling, because it explains the whole shape of the app. Bonnie's world vocabulary was once exactly the three applications she could watch. That quietly made "ambient context" a synonym for "things with eyes." A calendar read had no world to file itself under, so it reached nobody and was gone by the next turn — while an identical read from Pages persisted as a fact. The user-visible symptom was "she can pull documents but not calendars." The fix was to enumerate every actor completely and treat vision as an upgrade rather than an entry fee.
An assistant built that way does not need your screen most of the time. That is a privacy property you get from a taxonomy decision, not from a policy page.
OpenAI's Computer History reads the accessibility system too. It buffers those events on your Mac for up to 48 hours and then sends them to OpenAI's servers to be processed into memories. OpenAI says the events are not retained after that and are never used for training, and I have no particular reason to doubt it. But your intent left. Your keystrokes and your clicks made the trip, and the only thing between them and a different retention policy is a sentence on a docs page that a company can revise. Bonnie's read of the same tree is requested in-process, consumed in-process, and lives for the turn. Nothing about the accessibility API forced either outcome. Two apps read the identical structure off the identical system and one of them puts it on the wire.
That is the whole reason I keep saying the architecture is the product. The permission dialog looks the same in both apps.
And you can check her work. The Routes pane records what each turn actually used: the intent, the evidence, the reason a Skill was chosen over its alternatives, the package revision it ran under.
The trace has a hard floor written into it — raw selected text, generated prose, source code, and secrets never enter route traces or durable receipts. It is immutable history, and it is observational: nothing reads it on a decision path. It exists so that "why did she do that?" has an answer that is not a guess.
The rule: an assistant that can act on your machine owes you a legible reason for every action, and the reason must cost you no privacy to read.
Where the memory lives
Meta's app remembers you in a surface called "About Me," and that memory lives in your Meta account. Delete the account and the personalization goes with it. Switch assistants and you start from zero.
The memories Computer History builds come back down and stay on your machine as plain-text files — which sounds like the thing I have been arguing for, until you read the warning OpenAI puts in its own documentation:
Computer History files can contain sensitive information. They are not encrypted by Computer History, and other programs running as your macOS user may be able to access them.
Credit for the directness; that is a more honest sentence than most companies would ship. It also names a real hazard. A local file is only as private as the process list around it, the interaction events sit there unencrypted for their 48 hours, and the memory files stay until someone deletes them. Locality is not the same as safety.
Bonnie's memory is a program called Totem that runs on your Mac. It is a vector store and a knowledge graph, and it persists as ordinary files under ~/Documents/totem-db. Keeping this encrypted and decrypted into memory will be one of the goals. Although, I have an even better idea that will truly make a user feel their knowledge is theirs and theirs alone.
You can see the whole thing. The Totems pane browses the nodes, the library, the graph, the retrieval traces. You can rename an entity, merge two that turned out to be the same person, delete a relationship the extractor got wrong, and re-run extraction — each behind a confirmation, because it is irreversible surgery on your own memory. You can choose how entities get extracted at all: a hosted model, on-device MLX, or keywords only.
An alpha version of this idealogy was built into an iOS application earlier this year: https://seer.social. It was through this creation that I was able to understand some important trade-offs in the approach for the totem design.
The rule: personalization you cannot carry, inspect, or delete is not yours — it is a retention feature wearing your name.
Act, or advise
Here is the difference I care about most, and it is the one least visible from a feature list.
Meta's app reads your window and gives you a suggestion. You carry it out. The loop closes in your hands, which means the assistant is a very good advisor that never takes responsibility for a result.
OpenAI does not fit that binary, which is why it is the serious comparison. Computer Use acts — on your real Mac, with Screen Recording and Accessibility granted, clicking and typing in your actual applications. That is the same category of power Bonnie claims. So the question stops being does it act and becomes the better one: what does acting oblige it to show you, and what stops it when it is wrong?
OpenAI's answer to that is guidance. Keep sensitive apps closed unless the task needs them, review the permission prompts, reserve "Always allow" for apps you trust. That is sensible advice and it is entirely advice — it lives in a docs page, addressed to you, enforced by your attention.
Bonnie finishes the job. When she writes into your document, she is not printing text into a chat pane for you to copy — she is running a validated Ability through an adapter that types at your cursor in Scrivener, or edits the passage you named, or presses the control the page is offering. Every run leaves a receipt: the frozen package identity, the invocation, the effect, whether it was reversible.
That is only safe because of what sits under it. An Ability declares a Capability — its effect, its permissions, the classes of target it may touch, whether it needs confirmation. Those constraints are executable policy and a hard floor, not advice to a model. A package that tries to use a destructive capability without requiring confirmation does not load; the validator refuses it.
And the final authority is not the software:
The person always wins. After physical input cancels a run Bonnie does not reclaim focus or move the cursor back underneath them.
This is visible automation.
You watch it happen. You touch the keyboard and it stops. It does not fight you for the cursor.
The rule: an assistant permitted to act must be interruptible by a human hand, and must never be able to quietly undo that interruption.
Why open source is load-bearing here, not decorative
Bonnie is GPL-3.0. That matters for the obvious reason and for a much less obvious one.
The obvious reason: you can read what it sends. Nobody has to take my word about the accessibility tree, or about what is deposited to Totem, or about which model sees which bytes.
The less obvious reason is the extension model. Everything Bonnie knows how to do ships as a .bonnie package: canonical JSON, integrity-digested, optionally signed. Ten of them come bundled — Writing, Coding, Design, Sketch, Scrivener, Browsing, Chrome, Architect, Window Management, ShaderFeel. As I iterate through the ability design and implementation I aim for it to be easier for anyone to teach Bonnie an application of theirs. With a helpful front-end site that behaves as the ability studio, with helpful guides and examples.
Teaching Bonnie a new application is a declaration with no Swift in it at all. Scrivener support ships as scrivener.bonnie; there is no Scrivener source file anywhere in the repository. Ability Studio lets you open any package, fork it, edit it visually, validate it, and export it for someone else.
A flourishing open-source community that is centered around this new paradigm would be the successful outcome in my eyes. Another reason why I feel this interaction paradigm can only succeed if it is open-source in its purest sense.
Compare the trust question in each direction. To trust OpenAI's or Meta's app you have to trust the respective institution. To trust a Bonnie package you have to read a JSON file, and the runtime will refuse it if it lies about its own effects.
The space was already here
There is a name for what the industry decided this would require. Spatial computing was supposed to arrive as a new room — a headset, a fresh canvas, applications rebuilt as volumes, a boundary you cross to get in. The promise underneath it was a computer that knows where things are and what is around you. The price was leaving the desk you already work at.
But the space was already here. Your screen is a space. It has regions and containers, a focused document, a control under the pointer, a hierarchy of windows that know their own names — and macOS has been publishing a structured map of it for two decades, because blind users needed that map long before assistants did. The accessibility tree is not a workaround for lacking a camera. It is the floor plan. And the taxonomy that calendar bug forced on me — enumerate every actor, treat eyes as an upgrade rather than an entry fee — is what makes it a map of the whole building instead of a map of the three rooms that happen to have windows.
An assistant that reads that map knows where it is. It knows without a headset, without a screenshot, and without asking you to move. Bonnie does not open a new room for you to work in — she works in Scrivener, in the document you already have open, at the cursor where you left it, and gets out of the way the moment your hand moves.
Ambient clients is spatial computing, while maintaining the spaces that already exist.
That is the bet. It is not that a solo project out-features Meta. It is that when an assistant is this deep inside your work — reading your windows, typing into your documents, holding your notes — the question of who owns the thing stops being ideological and becomes practical.
Meta's app works for you, in the sense that a service works for you.
Bonnie is built to work with you: in parallel, on your hardware, out of a memory you own. The accessibility tree of macOS isn’t just a feature—it’s the Rosetta Stone for this entire vision. Apple didn’t just build an OS; they accidentally (or perhaps intentionally) mapped the DNA of human-machine interaction in a way that no one else has fully realized. The full argument for why that distinction deserves a name — and an architecture — is the longer piece next door: Ambient Intelligent Systems.