AI Orchestrator

After getting a local AI coding agent running in my home lab, the next experiment was obvious: instead of one model trying to do everything, build a team. I created several sub agents and gave them one boss. The orchestrator’s only job is to delegate. Under it sit a coder, a tester, and a designer, all running on a small 4B model on my RX 6600. Nothing leaves my network and it costs nothing to run.

The structure

Each agent is just a definition file: a role prompt plus an explicit tool list. The permissions are the design. Read-only agents can look, the coder alone can write, and the orchestrator can’t touch files at all:

# orchestrator
tools: delegate_task, read_file (backlog only)
rule: never write code, never edit files, hand out ONE task at a time

# coder
tools: read_file, grep, write_file
rule: one tiny change per task, match existing conventions

# tester
tools: read_file, run_command (scoped to the game dir)
rule: run it, report what happened, suggest nothing

Cutting each agent’s tool access also cuts the token overhead that eats small models alive, the same schema-bloat problem I hit with my Ansible agents. A 4B model with three tools stays coherent a lot longer than one carrying twenty tool definitions it mostly won’t use.

The loop

The test project was a small text adventure game, picked deliberately because it doesn’t matter. A design doc holds the backlog, top task first, one tiny change at a time:

## Backlog
1. Add a "look" command that reprints the room description
2. Items in rooms: pick up, drop, inventory list
3. Save/load to a JSON file

The orchestrator reads the top task, hands it to the coder, the tester runs the game and reports back, and the loop repeats. If the pattern breaks, it breaks on something that doesn’t matter. That’s why the toy project earns its place.

What the small models taught me

The biggest lesson: small local models fall apart when you hand them big asks. Give one a whole feature and it wanders off and breaks things. Give it one tiny change with clear conventions to follow and it usually gets there. Task size is the real tuning knob, more than temperature or prompts.

The second lesson was the escalation rule. If the coder fails the same task twice, the orchestrator stops retrying, logs the failure to my shared memory system, and the task gets flagged for a bigger cloud model:

on second failure:
  store_memory(topic="escalations",
    text="Task 3 (save/load) failed twice: JSON serialization of
          room objects. Needs bigger model or human.")
  mark task ESCALATED, move to next

Without that rule, a small model will happily burn an afternoon failing the same task forty ways. Same way a shop runs: the apprentice does the oil changes, and you don’t hand them a transmission rebuild just because they’re standing there.

The same pattern now does real work on my network automation. A read-only recon agent investigates how my Ansible repos are structured, then a writer agent drafts playbooks that match those conventions. The takeaway: orchestration isn’t about making one agent smarter. It’s about limiting what each agent can do so the whole team stays predictable.