Every conversation with an AI agent starts the same way: from zero. You explain your stack, your preferences, the decision you made last week and why. The agent nods, helps, and forgets all of it the moment the session closes. Next time, you do it again.
We've gotten very good at making agents smart. We've spent almost no effort making them remember. And memory, it turns out, is the boring infrastructure problem that quietly caps how useful any agent can actually be. A brilliant assistant with no memory is a stranger you have to re-onboard every single day.
So I built one. It's called MegaBrain, it's open source, and this is the short version of how it works and what I learned.
The shape of the problem
Here's the thing — my agents did have memory. ChatGPT kept its own memory of me. Claude kept its own, and managed it well. The problem wasn't absence. It was ownership. Each agent held a private version of me, locked inside its own walls, and I didn't own a single one of them.
Which means the memory doesn't move. The day a better agent shows up — and one always does — I'd start over. Every preference re-explained, every decision re-justified, the whole relationship rebuilt from zero, because the context I'd spent months building belonged to the tool, not to me. Switching agents shouldn't cost me my own history.
Meanwhile, I already had the one record that was mine: my Obsidian vault. Months of notes — project decisions, architecture trade-offs, half-formed ideas, the reasons behind choices I'd otherwise forget. Plain markdown files. The most honest record of how I think, and the only memory in this whole picture that I actually own.
The gap was that my agents couldn't read it. Or rather, they could read a file if I told them exactly which one — but that's not memory, that's me doing the remembering and the agent doing the typing. Real memory means the agent can ask "what did she decide about X?" and find the answer by meaning, across hundreds of notes, without me pointing the way.
So that's the problem, in two sentences: the memory my agents had wasn't mine, and it didn't move with me. And the memory that was mine, my agents couldn't read.
The build
The architecture is deliberately unglamorous:
Obsidian vault → GitHub → Sync Worker → Neon (pgvector) → MCP Server → Agents
A few decisions worth naming, because the interesting part of any build is the trade-offs you accept:
Markdown stays the source of truth. The vault lives in Git. MegaBrain never owns your notes — it indexes them. Your data is portable, version-controlled, and yours the day you decide to walk away. This is the whole point: the memory belongs to you, not to any model. A new agent shows up next year, it speaks MCP, you point it at the same brain — and it knows you on day one. No starting over. "Yours forever" isn't marketing; it's a design constraint I refused to break.
Embeddings carry the meaning. Keyword search finds words; memory needs meaning. So each note gets turned into a vector with OpenAI's text-embedding-3-small. The small detail that punches above its weight: I don't embed the raw note. I concatenate the title, the tags, and the content before embedding. A note titled "Why we dropped the PR workflow" with a #architecture tag gets found even when the body never repeats those words. Cheap trick, real recall improvement.
Search is cosine distance, not keyword matching. Postgres + pgvector does the heavy lifting — ORDER BY embedding <=> query_embedding and you get the closest notes by meaning. No Elasticsearch, no vector-DB-of-the-month. Just Postgres doing one more thing well.
One service, ~$5/month. Sync worker and MCP server run in the same Node process on Railway. Splitting them would be more "correct." It would also be more expensive and slower to ship. For a single-user system, correct-later beat correct-now.
MCP is the contract — and the memory is shared. The agent talks to the memory through six tools — search_brain, get_note, list_notes, add_note, update_note, delete_note. Because it speaks MCP, any compatible client works: Claude, ChatGPT, Cursor, Postman, whatever ships next. But the real unlock isn't "works with any agent" — it's that they all work with the same memory. ChatGPT writes a note, Claude reads it five minutes later, and both are operating on identical context. Your agents stop living in silos, each with its own half-picture of you. One brain, many faces. The memory outlives any single model — and it's shared across all of them at once.
The part I keep thinking about
Here's what surprised me. The hard problem wasn't the embeddings or the vector search — those are solved, well-documented, almost commodity now. The hard problem was plumbing: getting a vault to sync reliably, surviving iCloud's habit of mangling files mid-write, making incremental re-indexing trigger off a webhook so it's near-instant instead of a nightly batch.
That's the pattern with memory in general. It isn't a research frontier. It's an infrastructure frontier. And infrastructure problems get neglected precisely because they're not exciting — until the day everyone realizes their agents have been suffering from amnesia the whole time, and the people who quietly solved the boring problem are the ones with a head start.
I've made a habit of building these unglamorous pieces before they're obviously needed. Sometimes the rest of the world catches up and ships something similar. That's fine. The point was never to be first to a headline — it was to have working memory the day I needed my agents to actually know me.
Try it
MegaBrain is MIT-licensed and on GitHub. One setup script, a Neon free tier, and a Railway hobby plan get you running. If you live in Obsidian and you're tired of re-explaining yourself to your tools — to each of your tools, separately — it's built for you.
Your notes already are your memory. This just lets every one of your agents read it, write to it, and finally share the same picture of you.
MegaBrain is one of several open tools coming out of Controlled Mayhem. More lab notes as we ship them.