Backups that survive being restored
Every backup story ends at a restore. Until I have restored a file, a database and a whole machine from the archive, I assume there is no backup.
Three levels
Files: nightly archive, kept for two weeks. Database: dump plus binary log, so a point in time is reachable. Machine: a documented list of packages and units, because rebuilding by hand is faster than restoring an image that no longer boots.
Check the boring things
Timestamps, ownership, permissions. A restore that brings back files owned by the wrong user is a failure that shows up days later. I keep a small script that restores into a temporary directory and compares checksums with the source.
Offsite or nothing
A backup on the same disk is a snapshot, not a backup. It protects against a mistake, not against a dead machine.