Seer's Journey to Mary

I dropped a bunch of essays on August 16 this year, mostly generated summaries of all of my work this year. A majority of it was covered in Ambient Intelligent Systems. I generated them, because of time and rest, while still building the systems and architecting them mentally in parallel. That's one of the good things about A.I., as long as you proofread it. Which I am working on, but I prefer to be an editor at times. Although, I do like the prose sometimes — Ambient Intelligent Systems came out rather poetic, it felt like a journey, the emotion that went into it was documented. But I really do want to get technical, otherwise it's quite difficult to truly communicate how years of learning compounded into what I am building, MaryOS. Mary is this culmination of experiences integrated into one solution, with extreme care. An open source project I plan on supporting on the side while I work on Gita's Ballad. A project I see myself maintaining for a very long time.

Seer was how it started. I wanted to tackle the royalty problem of written works used in large language models. I initially went about the development on a high-level approach, to accrue potential funding in order to support the deep-dive. While working on the side, and maintaining so many parallel minds in one, it's quite tough to go deep, but MaryOS aligning everything in the end has helped me relax so much that it's now enjoyable to write about what my intentions are with this project. And funding does not seem to be that much of a worry with this much clarity of the future I have currently.

Seer was to be an embedding powered database tool that ingests documents, personal documents, at scale to personalize LLM outputs to a user. While applying a simple magnitude comparison of what context was retrieved with a generated output, using similarity strategies. It's not a robust solution, not at the model layer prior to token prediction, but it was enough to prove the point. I intend to get to the token prediction layer now with Mary, looking forward to writing about that in the future and how I approach watermarking predictions with their provenance as well.

I had built a base iOS application to really hone in my design thinking around Seer, it was honestly a great project overall to onboard me back from the self-directed research journey I went through since the LLM era began. It brought me back and it pushed me to get to this point today. I started in October 2025, planning my approach and plan since January to February 2025, and a year later, I am certain we can build proper copyright systems with generative technologies, we just haven't tried yet.

Seer is a Swift-on-Server application that has multiple routes, using Supabase for authentication, and was hosted on Scaleway. The key component was this storage solution I had derived on my own based on my knowledge of lookup tables (LUTs) that are used for filters in color grading technologies. Where a document can be partitioned into multiple segments, and each segment has an embedding vector that is stored in memory, compressed such that the entire 1024-dimension vector isn't always held in memory — only 64-dim chunks are — and when candidates are found, the disk is finally read to extract the full text associated with it. It worked really well, the partitioning got so specific that it almost felt like magic. And stitching multiple 64-dim searches back into what a realized 1024-dim search would look like delivered on trust in parity to source material. Especially when aggregating short-form posts or social media content from Bluesky.

I had gravitated towards a search system that was more robust and efficient, tackling HNSW initially — a way to store these vectors in neighborhoods such that it could jump to a neighboring document without a full linear scan of everything stored in memory. These neighbors were created at ingestion, but I soon realized the maintenance of an HNSW was very tedious and rebuilding graphs upon deletion and insertion was costly. Which is not a problem per se, but I wanted to explore more possibilities prior to investing too much time in building a better way to maintain HNSWs for large-scale applications as a solo engineer.

Prior to this, I always had this thought, "Decentralized intelligence centralizes growth for the whole." It also stems from one of my first jobs, where our slogan was "Big brains, small devices." In 2023, I always felt that the future was about millions of small models aggregating their specialized knowledge into a singular aggregated understanding, and now I decided to change paths in February to March 2026 to stick to that.

I wanted the storage portion of this solution to be distributed, Seer connects to these as nodes for retrieval, and not only is the stress of maintaining a large navigable world in the cloud of multiple users absolved, it also solves the grounding of personalization at its core. Users own their data and decide to share it with the network or not. So then, Totem came right after, splitting the codebase into two strict groups. Seer is the server API layer to handle on-device and LLM APIs, while storage is solved via totem, connecting and searching for totems via gRPC, streaming the responses directly into the server when needed.

I wanted to build Totem as an option to be on an extendable drive from the start, so I bootstrapped an old computer to host Ubuntu 24, got Swift on Linux working, spawned up the totem and got it to connect to my MacBook's hosted Seer via a local IP. It worked amazingly and it further cemented this wonderful idea of modularity, decentralized thought, freedom, and personalization. Transitioning from HNSW to knowledge graphs, this idea of strict entity and relationship bindings created this search functionality that did not feel random, it was deterministic, but a question that soon came after was how to generate these schemas on the fly rather than manually adjust them.

I aim to dive deeper into all of these approaches later on, viewing this initial writing as an introduction and a map for the revised Ambient Intelligent Systems write-up, because journeys are important to document. Otherwise we might forget the memories, or someone will try to rewrite them.

So yeah, started with royalty research, transitioned into distribution, then came the search for the vessel. A chat application never felt like a goal or end product of this era. It always felt like a demo to me. Not only that, since 2022, generative tech made me feel my creativity was worthless, the process of creation was stolen by a machine in a box with leaders that mask their intentions without providing us security in our futures. It was an antithesis to my humanity, but A.I. also brings good, it brings speed, it saves time. I wanted to bridge these two realities. So I thought of this idea of ghosts, spirits, angels that appear to help inspire my work when I am undergoing a creative process. This idea of a parallel helper, a pair programmer. A personal AI operating system.

So then came Bonnie. I liked this idea of "Bonnie & Clyde," not in a romantic sense, but in a collaborative light to solve problems together. Tackling the computer use problem, which got coined soon later. Proving the human-computer interaction was everything I asked for. Being able to navigate to YouTube videos hands-free, play Apple Music while she understood where I was working on in Xcode helping me find things.

Not coding for me, but actively helping me remain focused.

Then this spark came, when she referenced Gita's Ballad out of the blue, it related to what I was seeing, but it wasn't within that context.

That was the Ambiance I was reaching for. And soon, it will be released as MaryOS, the polish, the kohinoor. The one who will carry this vision in its entirety as a new way to operate our machines.