I use a few different AI assistants. Claude for development work, ChatGPT in the browser, and local models running on my own machine. They’re all useful, but every one of them has amnesia. Worse, they have separate amnesia: I’d figure something out in one tool and the others had no idea it ever happened. So I built them one shared memory.
The pieces
A ChromaDB vector database runs in a Docker container, and a small Python layer around it gives every assistant the same four operations: store a memory, search memories, list recent ones, check status. The database side is one service:
services:
chromadb:
image: chromadb/chroma:latest
ports:
- "8000:8000"
volumes:
- chroma_data:/chroma/chroma
volumes:
chroma_data:The AI tools connect through MCP, basically a standard plug that all of them speak now. The server side is shorter than people expect. The whole store/search core fits in a screen:
from mcp.server.fastmcp import FastMCP
import chromadb, datetime, os
mcp = FastMCP("shared-memory")
client = chromadb.HttpClient(host=os.environ.get("CHROMA_HOST", "localhost"))
col = client.get_or_create_collection("memories")
@mcp.tool()
def store_memory(text: str, agent: str = "unknown", topic: str = "general") -> str:
"""Save a fact or decision that any assistant can recall later."""
doc_id = os.urandom(6).hex()
col.add(documents=[text], ids=[doc_id],
metadatas=[{"agent": agent, "topic": topic,
"stored": datetime.datetime.utcnow().isoformat()}])
return f"stored {doc_id}"
@mcp.tool()
def search_memory(query: str, max_results: int = 5) -> str:
"""Semantic search across all stored memories."""
r = col.query(query_texts=[query], n_results=max_results)
return "\n\n".join(r["documents"][0])Anything that can’t do MCP still gets in through a command-line script that hits the same collection. Search is semantic, so I can ask “what IP did we put that service on?” and it finds the answer even if it was stored as part of an old conversation about something else.
Feeding it history
The best part was ingesting the past. I exported old chat sessions, ran them through an ingest script that chunks and stores them, and suddenly a little 4B model running on my RX 6600 could answer questions using research I did months ago in a browser. One thing I learned quick: storing the three key takeaways from a session beats dumping the whole transcript. Chunked raw transcripts pollute search results with filler; curated summaries hit. Curated beats raw.
What broke
My first local model would confidently announce it was going to use the memory tool and then just… stop. Turned out some models simply aren’t trained for tool calling, and swapping to one that was fixed most of it. The other gotcha was context size: my chat had quietly overflowed the model’s context window, which pushed the tool definitions right out of its view. Same as diagnosing a car. The symptom was “model is dumb,” but the actual cause was two layers down.
Portable on purpose
The code is a git repo with a setup script, the data lives in a Docker volume I can back up and move, and the server address is one environment variable. When it eventually moves off my desktop onto a proper server, nothing gets rebuilt, the clients just get a new CHROMA_HOST.
One Docker container and a few hundred lines of Python, and my AI tools finally remember what we did last week. All of it running local, on hardware I already own.