Homelab Monitoring Without the Guesswork

My lab monitoring is a Telegraf, InfluxDB, and Grafana stack in Docker. Telegraf polls the switch and firewall over SNMP every 60 seconds, InfluxDB stores it, Grafana draws it. Standard stuff, but a few decisions made it much less painful than most write-ups make it look.

Skip the MIB circus, use numeric OIDs

The advice I’d give loudest: put numeric OIDs in the Telegraf config instead of named MIBs. Getting MIB files installed and resolving inside a container is its own hobby, and the numeric OIDs just work everywhere, forever. This block gets uptime plus the full interface table with traffic, status, and errors, no MIB installed anywhere:

[[inputs.snmp]]
  agents = ["192.168.1.2"]        # the switch
  version = 2
  community = "$SNMP_COMMUNITY"   # from the env file, not hardcoded
  interval = "60s"

  [[inputs.snmp.field]]
    name = "uptime"
    oid = "1.3.6.1.2.1.1.3.0"     # sysUpTime

  [[inputs.snmp.table]]
    name = "interface"
    inherit_tags = ["hostname"]
    oid = "1.3.6.1.2.1.31.1.1"    # ifXTable: 64-bit counters

    [[inputs.snmp.table.field]]
      name = "ifName"
      oid = "1.3.6.1.2.1.31.1.1.1.1"
      is_tag = true

  [[inputs.snmp.table]]
    name = "interface_state"
    oid = "1.3.6.1.2.1.2.2"       # ifTable: operStatus, errors

Use ifXTable for traffic counters, not the old ifTable ones. The 32-bit counters in ifTable wrap in under an hour on a busy gigabit port and your graphs turn into sawtooth nonsense.

Everything provisioned from files

Grafana datasources and dashboards load from YAML and JSON in the repo, bind-mounted into the container:

# grafana/provisioning/datasources/influx.yml
apiVersion: 1
datasources:
  - name: InfluxDB
    type: influxdb
    url: http://influxdb:8086
    jsonData:
      version: Flux
      organization: homelab
      defaultBucket: telegraf
    secureJsonData:
      token: $INFLUX_TOKEN

Before this, my dashboards lived in a Docker volume, which means they lived nowhere. If it’s not in git, one bad container update and you’re rebuilding panels from memory. Secrets moved out of the compose file into a .env that stays out of the repo, and the compose file just references them:

services:
  telegraf:
    image: telegraf:latest
    env_file: .env
    volumes:
      - ./telegraf/telegraf.conf:/etc/telegraf/telegraf.conf:ro

Verify the pipe, not the green checkmark

When the first dashboard looked broken, sparse points scattered where lines should be, I went digging for the fault and there wasn’t one. Telegraf polls every 60 seconds and the panel was zoomed tight enough to show the gaps between polls. Zoom out or set the panel’s minimum interval to your polling interval and the lines come back.

The related habit that’s saved me since: verify the datasource with a real query instead of trusting the UI’s connection test. The green checkmark only proves Grafana can reach the port:

curl -s -H "Authorization: Token $INFLUX_TOKEN" \
  "http://localhost:8086/api/v2/query?org=homelab" \
  -H 'Content-Type: application/vnd.flux' \
  -d 'from(bucket:"telegraf") |> range(start: -5m) |> limit(n:5)'

If that returns rows, data is flowing regardless of what any dashboard says. It’s the monitoring equivalent of checking fuel pressure with a real gauge instead of reading the dash light.

The graphs are not the point. The point is that when something acts up, I look at history instead of guessing, the same way a scan tool beats listening to an engine and nodding.