Giving My AI Tools a Shared Memory
I use a few different AI assistants. Claude for development work, ChatGPT in the browser, and local models running on my own machine. They’re all useful, but every one of them has amnesia. Worse, they have separate amnesia: I’d figure something out in one tool and the others had no idea it ever happened. So I built them one shared memory.
The setup is simpler than it sounds. A ChromaDB vector database runs in a Docker container on my machine, and a small Python layer around it gives every assistant the same four operations: store a memory, search memories, list recent ones, check status. The AI tools connect through MCP, basically a standard plug that all of them speak now, and anything that can’t do MCP can still use a command-line script. Search is semantic, so I can ask “what IP did we put that service on?” and it finds the answer even if it was stored as part of an old conversation.
The best part was feeding it history. I exported a bunch of old chat sessions, ran them through an ingest script that chunks and stores them, and suddenly a little 4B model running on my RX 6600 could answer questions using research I did months ago in a browser. One thing I learned quick: storing the three key takeaways from a session beats dumping the whole transcript. Curated beats raw.
Not everything worked the first time. My first local model would confidently announce it was going to use the memory tool and then just… stop. Turned out some models simply aren’t trained for tool calling, and swapping to one that was fixed most of it. The other gotcha was context size: my chat had quietly overflowed the model’s context window, which pushed the tool definitions right out of its view. Same as diagnosing a car. The symptom was “model is dumb,” but the actual cause was two layers down.
It’s like a shop manual that writes itself. Every job adds pages, and whichever tool I pick up next has already read them.
The whole thing is portable on purpose: the code is a git repo with a setup script, the data lives in a Docker volume I can back up and move, and the server address is one environment variable. When I eventually move it off my desktop onto a proper server, nothing gets rebuilt.
One Docker container and a few hundred lines of Python, and my AI tools finally remember what we did last week. All of it running local, on hardware I already own.