I’ve been running a home lab since 2002, but the last few weeks it’s turned into something closer to a full enterprise analog: NetBox as the source of truth, Ansible with vault-encrypted credentials, Semaphore running playbooks from Git, automated encrypted config backups, and a Telegraf/InfluxDB/Grafana monitoring stack. Self-hosted GitLab with a GitHub mirror for version control. The newest piece: a fully local AI coding agent. No cloud, no API costs, nothing leaving my network.
The GPU reality check
Running this on consumer hardware taught me a few things fast. My GPU is an RX 6600, which AMD does not support in ROCm, so the compute path most LLM guides assume simply doesn’t exist here (community patches are starting to change that, but I wanted stable, not experimental). Vulkan turned out to be the fallback that just works: LM Studio and llama.cpp both ship Vulkan backends, and picking the Vulkan runtime got the model onto the GPU with zero driver surgery.
One diagnostic gotcha worth passing on: Windows Task Manager’s default GPU graph shows the 3D engine, which sits near zero during inference and makes you think the GPU is loafing. The real number hides in the Performance tab under the Compute dropdown. I spent time chasing a “CPU-only” problem that didn’t exist. And if you run models under WSL2, closing the terminal doesn’t return the RAM; the vmmem process holds it until you actually run wsl --shutdown.
The token overhead problem
The bigger problem is invisible until you measure it: agentic coding tools send their full tool schema with every single request. Every file tool, every search tool, every edit tool, described in JSON, on every message. On a small local model that overhead is brutal, roughly a 10k token prompt before your question even gets read, and small models get noticeably dumber as the context fills.
The fix was scoped agents. Instead of one agent with every tool, I define narrow ones, and the read-only ones carry a fraction of the schema:
# agent: repo-recon (read-only)
tools: read_file, grep, list_dir
purpose: answer questions about how the Ansible repos are structured
# agent: playbook-writer (write-enabled)
tools: read_file, grep, write_file
purpose: draft playbooks matching existing conventions, nothing elseCutting the tool surface cut the prompt overhead dramatically, and it doubles as a safety rail: the agent that answers “how does our inventory work?” physically cannot edit anything.
What it’s actually good for
End result: I can ask an agent questions about my Ansible repos or have it draft playbooks matching my existing conventions, entirely offline, at around 5 tokens a second on hardware I already owned. That speed is near the ceiling for this card, not a config problem, so the workflow adapted instead: local model for boilerplate drafts and read-only exploration, cloud model for correctness review before anything touches real infrastructure.
It’s slower than cloud models and I wouldn’t trust it unsupervised near production, but that was kind of the point. I wanted to understand the limitations behind “AI coding agent” before trusting one near real infrastructure.