Ambient Abilities

A field report from the authoring surface of the ambient assistant, and the routing question underneath it.

The assistant I've been writing about on this blog has a new name. Bonnie is now Mary — a re-architecture, not a rebrand: the same commitments (accessibility tree as ground truth, memory the user owns, actions that leave a receipt) cut down to their load-bearing shape. Most of what changes with the rename is internal. What doesn't change is the bet underneath it: teaching Mary a new application should never require writing Swift.

This idea lives in one screen, the Ability Studio, and I want to walk through it in the order I've actually been living in it lately — a recipe, the ability it belongs to, one skill inside that ability, and then the part I think is the most interesting engineering in the whole app: the tool that lets me rehearse how a sentence gets routed, before it ever reaches a model.

A recipe is a macro with an argument

This is "Pair Program," a recipe I built to review a diff without opening a terminal:

A Recipe chains Skills into one step Mary can be asked for directly — this ability's own skills, or any installed ability's. "Pair Program" reads the current file, reads the symbol under the cursor, pulls the project outline, then hands off to Mary herself to plan the minimal change before "Coding" completes it and runs a build check, and finally Mary explains what changed. Two of those seven steps aren't bound to an adapter at all — they're owned by Mary, not by an ability, which is the recipe's way of saying "reasoning happens here, mechanically bound tool calls happen everywhere else." A recipe is not a script that only calls tools; it's a sequence that can hand control back to the model and take it back again.

It's also tagged Branching, and the caption says the quiet part out loud: this recipe recovers on failure, so its order is edited in Advanced. The steps you see aren't a strict top-to-bottom pipeline — they're a graph with recovery edges, and the linear list is the readable projection of it. And at the bottom, a recipe declares its own contract the same way a skill does: task · string · required. That detail matters more than it looks — a recipe isn't documentation of a procedure, it's an invocable capability with a typed signature, indistinguishable from a single skill to whatever calls it.

Abilities are the unit you actually teach

Zoom out one level and you get the screen the recipe lives inside — the Ability Studio, open on the Apple Music ability:

An Ability is a portable family of behavior and operating policy. It declares a paradigm — Apple Music is applicationExpertise, one of four (the others being discipline, systemControl, and reasoning) — and it can extend another ability. Apple Music extends Multimedia, a discipline: the general shape of "something that plays media" is defined once, and Apple Music realizes it for one specific app. The sidebar shows the rest of the roster living under the same rule — Coding and Writing as disciplines, Chrome and Safari and Scrivener as application expertise, Window Management as system control.

The right-hand Tune panel is the part worth sitting with, because it looks like documentation and isn't. Summary, Listens for, Phrases, Not when, Also called, Sounds like — every one of those fields is bounded, structured data that becomes part of the ability's embedding corpus. There's no free-text system prompt to hand-tune here and no fine-tune to kick off. Teaching Mary that "in the music app" means Apple Music is the same kind of edit as adding a row to a spreadsheet. That's what makes an ability something a non-engineer can extend: the surface you're allowed to touch is exactly the surface that has semantic reach, nothing more, nothing hidden underneath it.

Below that, the Skills row makes the composition explicit: Apple Music realizes what it extends — Say What's Playing, Control Playback, Search the Catalog, Play a Song, Open the Player — then the tree continues past the ability's own boundary: Multimedia is what it extends, Window Management is what it's supporting it with, seven skills' worth. Vertical composition (extends) and horizontal composition (supporting) are drawn as the same kind of line, because to the router they resolve the same way. And the footer is honest about the state of things without being alarming about it: Active · 4 advisories · saving creates a local override. Editing a bundled ability never mutates the shared original — it forks into something local, and the advisories tell you what's incomplete before you find out at runtime.

A skill is the atomic thing that actually runs

One level further down, a single skill card, opened for inspection:

Say What's Playing is now_playing, and it's marked READY. Its tags are its actual contract: acts, runs unasked, Media Surface, from Multimedia. "Runs unasked" is a declared property, not a guess the model makes at inference time — it means this specific skill is allowed to speak up without being prompted, which is the difference between an assistant that answers questions and one that's actually ambient. "Media Surface" names the generic adapter underneath it: the same compiled provider that answers "what's playing" for Apple Music also answers it for every other app that presents transport controls the accessibility tree can see, which is the entire reason Multimedia gets to be a discipline instead of five copies of the same skill. You can drop this exact card into a recipe with one click, or jump straight to the Multimedia ability it's actually implementing. Nothing about the skill card is a preview of behavior — it's the behavior's own metadata, inspectable because it was never anything else.

The problem none of this solves by itself

Abilities and Skills give you a vocabulary. They don't, on their own, answer the question every computer-use architecture eventually has to answer: given what the person just said, which of these hundred things do you run?

The obvious answer — hand the model a system prompt listing every tool and let it decide, every single turn — is the one most agent frameworks ship with, and it has a cost that compounds with the size of the roster. Every turn pays for a full model round just to figure out where to go, before any actual reasoning happens. And the decision is only as legible as whatever the model's chain-of-thought happened to say that time, which is to say: not very. As Mary's ability roster grows, that path gets slower and less auditable at exactly the rate you'd want it to get faster and more auditable.

Mary's answer is to give the router its own opinion, separate from the model's, built from the same authored data you saw in the Tune panel. Every ability and every skill scores an incoming sentence against two tiers — an ability tier (tokens, phrases, aliases, exemplars) and a skill tier (title, summary, invocation, eligibility phrases) — and if one candidate clears a floor and beats everything else by a margin, Mary dispatches it with no model round at all. The shortcut is real work skipped, not work hidden.

Routing rehearsal: watching the router think

This is the tool that makes that claim checkable instead of asserted, run against the sentence "Play the RAO playlist":

The header comment on the file behind this screen says the important thing directly: not a simulation. The rehearsal sheet doesn't model what the router would do — it calls the exact same functions the turn loop calls, against the registry that's active right now. The registry 41398F89 stamp is the receipt of that: indexes are built once at reload, so a change you make here is inert until you save and the registry rebuilds. Rehearse before Save and you're testing the router that's actually running your Mac, not a copy of it.

Two thresholds run this screen. The floor, 0.62, is the minimum affinity for an embedding to have an opinion worth acting on — below it, a candidate is off the board, not just unlikely. The margin, 0.04, is how much clear air the leader needs over the runner-up before Mary trusts the embedding enough to skip the model, and both apply at both tiers. For "Play the RAO playlist," play_playlist clears the floor at 0.76 with shuffle_playlist ten points behind — well outside the margin — and multimedia does the same at the ability tier, 0.81 against a field that never reaches 0.62 at all. Two clean tiers, one winner each: "Mary answers with Play a Playlist, no model round at all."

The right panel is why this matters past Mary specifically. When two skills inside one ability start crowding each other, the instinct is to tighten the ability's phrases — the field you were just in on the Tune panel — but phrases, aliases, and exemplars feed the ability tier only, and two skills sharing an ability share that signal by construction, so no amount of ability-level tuning reaches them. The only lever that does is a fixture: an authored sentence written straight into that one skill's own corpus, the skill tier's actual data rather than a proxy for it. That's the shape I'd want other computer-use architectures to borrow — a cheap, authored, inspectable layer for the routing you can specify in advance, the model held in reserve for whatever it can't resolve, and a rehearsal loop that shows you which lever has reach before you spend an afternoon pulling the wrong one.