A green backup job is encouraging, but it is not proof that you can recover. It only proves that one command completed under one set of conditions.
A useful backup design is a chain: capture the application in a consistent state, validate the result, retain the right history, keep a copy outside the failure domain, and prove that the material can be restored. If any link is missing, the recovery plan is weaker than it looks.
Start with the recovery question
Before choosing a tool, ask what you are trying to recover. A database needs a restore point that the database understands. A virtual machine or volume backup can restore the surrounding operating system, service configuration, permissions, and application data. Those solve different problems.
For many self-hosted services, keeping both is sensible. Use an application-native export or snapshot for a clean, portable restore. Use a guest or volume backup for a faster way to bring back the whole service boundary. Then keep at least one independent copy outside the host or storage system you are protecting.
Validate before calling it a backup
Write backup output to a temporary name first. Run the format-specific check. Only then rename it into the published backup set. That keeps an interrupted archive or failed export from being treated as the newest good copy.
The right validation depends on the data. It may be listing an archive, running a database integrity check, checking snapshot metadata, or restoring into an isolated temporary target. A non-empty file is better than nothing, but it is not a restore test.
Retention is part of the design
Retention is not just a storage setting. It decides how far back you can go when corruption or an operator mistake is discovered late. Keep enough generations to cover the failure patterns you care about, and make cleanup narrow enough that it cannot remove unrelated files.
A safe cleanup rule targets one known backup family in one known directory. Broad recursive cleanup, wildcard paths, and assumptions about generated filenames are how a backup job becomes a deletion job.
Keep copies outside the failure domain
A valid archive stored on the failed machine is not disaster recovery. It may still help with a quick rollback, but it does not protect against host loss, storage failure, ransomware, or an accidental destructive action that reaches the backup directory too.
An independent copy should have a separate failure boundary. That might mean another system, different storage, offline media, or a provider with separate credentials and retention. The best choice depends on the service and budget. The principle is the same: one failure should not remove both the live data and every recovery option.
Test the restore, not the schedule
The most valuable backup exercise is restoring into an isolated target. Do not test by overwriting the live service. Build a disposable destination, restore the newest valid material, start the service, and check the data that matters.
That test exposes the dependencies people forget: encryption keys, certificates, service accounts, database versions, configuration, DNS assumptions, and recovery order. Document those dependencies without putting secret values in the document.
Common ways backup plans fail
- The timer exists, but the first manual run was never checked.
- The service account cannot write to the intended location.
- The filename produced by the tool differs from the one the validation step expects.
- Retention deletes partial output or another service’s archives.
- The off-host copy has never been checked after a restore.
- The only credential required for recovery is trapped inside the failed system.
- The documentation describes a plan that was never deployed or tested.
A simple recovery drill
- Create a known-good backup and validate it.
- Create more copies than the retention rule keeps, then confirm only the intended oldest copy is removed.
- Corrupt a copy and confirm the validation step rejects it.
- Restore the newest good copy into a temporary environment.
- Write down every non-secret dependency needed for that restore.
That drill is small enough to run before an emergency and useful enough to expose a weak link. The goal is not to collect the most backup files. It is to make recovery boring when the day comes that you need it.