Future Vision

Seer's Journey to Mary

September 4, 2026rao-studios/AIIseertotemmaryembeddingshnswknowledge-graphsdecentralization

I dropped a bunch of essays on August 16 this year, summaries of all of my work this year drafted with A.I. from my own notes and codebases. A majority of it was covered in Ambient Intelligent Systems. I drafted them that way to save time and rest, while still building the systems and architecting them mentally in parallel. That's one of the good things about A.I., as long as you edit what it gives you — and I'm still editing, because I prefer to be an editor at times. Although, I do like the prose sometimes — Ambient Intelligent Systems came out rather poetic. It felt like a journey, and the emotion that went into it was documented. But I really do want to get technical; otherwise, it's quite difficult to truly communicate how years of learning compounded into what I am building, MaryOS. Mary is this culmination of experiences integrated into one solution, with extreme care. A project I plan on supporting on the side while I work on Gita's Ballad, now in a private repository. A project I see myself maintaining for a very long time.

Seer was how it started. I wanted to tackle the royalty problem of written works used in large language models. I initially approached the development at a high level, to see whether the idea could earn support for a deeper dive. Going deep is hard when a project lives alongside everything else, but MaryOS aligning everything in the end has helped me relax so much that it's now enjoyable to write about what my intentions are with this project.

Seer was to be an embedding-powered database tool that ingests documents, personal documents, at scale to personalize LLM outputs to a user, while applying a simple magnitude comparison of what context was retrieved with a generated output, using similarity strategies. It's not a robust solution, not at the model layer prior to token prediction, but it was enough to prove the point. I intend to get to the token prediction layer now with Mary, and I look forward to writing about that in the future and how I approach watermarking predictions with their provenance as well.

I had built a base iOS application to really hone my design thinking around Seer. It was honestly a great project overall to onboard me back from the self-directed research journey I had been on since the LLM era began. It brought me back and it pushed me to get to this point today. I started in October 2025, after planning my approach from January to February 2025, and a year later, I am certain we can build proper copyright systems with generative technologies; we just haven't tried yet.

Seer is a Swift-on-Server application that has multiple routes, using Supabase for authentication, and was hosted on Scaleway. The key component was this storage solution I had derived on my own based on my knowledge of lookup tables (LUTs) that are used for filters in color grading technologies. Where a document can be partitioned into multiple segments, and each segment has an embedding vector that is stored in memory, compressed such that the entire 1024-dimension vector isn't always held in memory — only 64-dim chunks are — and when candidates are found, the disk is finally read to extract the full text associated with it. It worked really well. The partitioning got so specific that it almost felt like magic. And stitching multiple 64-dim searches back into what a realized 1024-dim search would look like delivered on trust in parity to source material. Especially when aggregating short-form posts or social media content from Bluesky.

I had gravitated towards a search system that was more robust and efficient, tackling HNSW initially — a way to store these vectors in neighborhoods such that it could jump to a neighboring document without a full linear scan of everything stored in memory. These neighbors were created at ingestion, but I soon realized the maintenance of an HNSW was very tedious and rebuilding graphs upon deletion and insertion was costly. Which is not a problem per se, but I wanted to explore more possibilities prior to investing too much time in building a better way to maintain HNSWs for large-scale applications as a solo engineer.

Prior to this, I always had this thought, "Decentralized intelligence centralizes growth for the whole." It also stems from one of my first jobs, where our slogan was "Big brains, small devices." In 2023, I always felt that the future was about millions of small models aggregating their specialized knowledge into a singular aggregated understanding, and now I decided to change paths in February to March 2026 to stick to that.

I wanted the storage portion of this solution to be distributed. Seer connects to these as nodes for retrieval, and not only is the stress of maintaining a large navigable world in the cloud of multiple users absolved, but it also solves the grounding of personalization at its core. Users own their data and decide to share it with the network or not. So then, Totem came right after, splitting the codebase into two strict groups. Seer is the server API layer to handle on-device and LLM APIs, while storage is solved via Totem, connecting and searching for totems via gRPC, streaming the responses directly into the server when needed.

I wanted to build Totem as an option to be on an extendable drive from the start, so I bootstrapped an old computer to host Ubuntu 24, got Swift on Linux working, spawned up the totem and got it to connect to my MacBook's hosted Seer via a local IP. It worked amazingly, and it further cemented this wonderful idea of modularity, decentralized thought, freedom, and personalization. Transitioning from HNSW to knowledge graphs, this idea of strict entity and relationship bindings created this search functionality that did not feel random; it was deterministic. But a question that soon came after was how to generate these schemas on the fly rather than manually adjust them.

I aim to dive deeper into all of these approaches later on, viewing this initial writing as an introduction and a map for the revised Ambient Intelligent Systems write-up, because journeys are worth documenting while the memory is fresh. Otherwise the details fade.

So yeah, started with royalty research, transitioned into distribution, then came the search for the vessel. A chat application never felt like a goal or end product of this era. It always felt like a demo to me. Not only that, since 2022, generative tech made me feel my creativity was worthless, as if the process of creation had been taken over by a machine in a box whose workings I couldn't see into, offering little security about our futures. It was an antithesis to my humanity, but A.I. also brings good: it brings speed, it saves time. I wanted to bridge these two realities. So I thought of this idea of ghosts, spirits, angels that appear to help inspire my work when I am undergoing a creative process. This idea of a parallel helper, a pair programmer. A personal AI operating system.

So then came Bonnie. I liked this idea of "Bonnie & Clyde," not in a romantic sense, but in a collaborative light to solve problems together. Tackling the computer use problem, before it had that name. Proving the human-computer interaction was everything I asked for. Being able to navigate to YouTube videos hands-free, play Apple Music while she understood where I was working in Xcode and helped me find things.

Not coding for me, but actively helping me remain focused.

Then this spark came, when she referenced Gita's Ballad out of the blue. It related to what I was seeing, but it wasn't within that context.

That was the Ambiance I was reaching for. And soon, it will be released as MaryOS, the polish, the kohinoor. The one who will carry this vision in its entirety as a new way to operate our machines.