How to test your backups (and why checking they ran isn't testing)

Here is the uncomfortable question: when did you last restore one of your backups?

Not "check that the cron job ran." Not "confirm the file landed in the bucket." Not "see a green tick in a dashboard." Actually restore one, into something real, and use the result.

If the answer is "never," you don't have backups. You have a folder of files you hope are backups, and hope resolves into fact at the worst possible moment. The GitLab database incident of 2017 remains the canonical case: five separate backup mechanisms, and when they needed one, none of them worked. Nobody had checked, because everything reported success.

What a green cron job actually proves

0 2 * * * /opt/backup/run.sh >> /var/log/backup.log 2>&1

Exit code zero tells you the last command in that script returned zero. That is all. Every one of the following produces a clean exit and a useless backup:

pg_dump connected to the wrong database and dumped an empty one
The dump succeeded, and the disk filled during the upload, truncating the file
A schema migration added a table the script's explicit table list never included
The database moved to a new host months ago and the script has been dumping a stale replica
Client-side encryption is on and nobody can find the keyfile
The archive is fine and the restore needs a tool nobody has installed
Permissions in the restored files don't match the UID the app runs as

Notice the pattern: none of these are backup failures. They are restore failures, and restore failures are invisible from the backup side. The only thing that surfaces them is a restore.

Four levels of confidence

Not every check is a drill, and it helps to be honest about what each one buys.

Level	What you do	What it proves
1. It ran	Exit code, log line, monitoring ping	A script finished
2. It's plausible	File exists, size in expected range, header parses	You have a non-empty file
3. It's intact	Checksums verify, archive lists cleanly	The bytes are the bytes
4. It restores	Load into a scratch environment and use it	You can actually recover

Most teams live at level 1 and believe they are at level 4. Levels 2 and 3 are cheap and worth automating on every run — but they are pre-flight checks, not the flight. A dump of an empty database passes all three.

Level 3 is where a good backup tool earns its place: restores that verify chunk integrity against expected checksums before reassembly catch silent corruption at drill time rather than at incident time. It is still not level 4.

The drill protocol

A restore drill has five parts, and skipping the last two is how drills become theatre.

1. Restore into somewhere disposable

Never over live data. A scratch directory, a throwaway container, a fresh database name — anything you can delete afterwards without thinking.

import { mkdtempSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";

const target = mkdtempSync(join(tmpdir(), "restore-drill-"));
await client.restore(snapshotId, target, {
  onProgress: (e) => console.log(`[${e.phase}] ${e.current}/${e.total}`),
});

The single most common way a drill causes an outage is someone restoring "just to check" over the thing that was working. Make the scratch target structural, not a matter of remembering.

2. Load it into a real engine

Files on disk are not a recovered database. Load them:

createdb app_db_restore
pg_restore --dbname=app_db_restore "$TARGET/db-dumps/app.dump"

This is where missing extensions, version mismatches, and absent roles surface — all of them things that would otherwise surface during an incident, at 3am, with an audience.

3. Verify the data, not the file

Row counts, then actual rows:

psql -d app_db_restore -c "SELECT count(*) FROM users;"
psql -d app_db_restore -c "SELECT * FROM orders ORDER BY created_at DESC LIMIT 5;"
psql -d app_db_restore -c "\dt"

Compare against production. A restore that returns 40% of the expected rows is a failure that a file-size check would have passed.

Check the schema too. A restore that is a migration behind is a restore that your application will not start against.

4. Use it

Point the actual application at the restored data and load a real page. Not a health check — a page that reads user data.

This is the step that catches file ownership mismatches, missing environment variables, absent uploaded files that live outside the database, and the search index nobody remembered was a separate system. A database that opens is not an application that works, and users experience the second one.

5. Time it, and write the number down

Start a stopwatch at step 1, stop it when step 4 succeeds. That wall-clock number is your real RTO — see RPO vs RTO for what to do with it.

Keep a log across drills. The number drifting upward as data grows is the early warning you want; discovering it during an outage is not.

If the measured number is four hours and your team believes "we'd be back in twenty minutes," you have just learned the most valuable thing this exercise produces. That gap is a real business risk that nobody knew existed.

Automating it

A quarterly drill that depends on someone remembering will happen twice and then stop. Put it in CI.

import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { execSync } from "node:child_process";
import { BackupClient } from "@lighthouse-web3/baas-js-sdk";

const client = new BackupClient({
  apiKey: process.env.LH_API_KEY,
  workspaceId: process.env.LH_WORKSPACE_ID,
});

const started = Date.now();
const target = mkdtempSync(join(tmpdir(), "drill-"));

try {
  // Always drill the newest snapshot — that is the one you would reach for.
  const { snapshots } = await client.listSnapshots("", 1);
  const [latest] = snapshots;
  await client.restore(latest.snapshotId, target);

  execSync(`pg_restore --dbname=drill_db "${target}/db-dumps/app.dump"`);

  const rows = Number(
    execSync(`psql -tA -d drill_db -c "SELECT count(*) FROM users;"`).toString(),
  );

  // A threshold, not just "greater than zero". An empty dump is the failure
  // mode this whole exercise exists to catch.
  if (rows < 1000) {
    throw new Error(`Only ${rows} users restored — expected at least 1000`);
  }

  console.log(`Drill passed: ${rows} users in ${(Date.now() - started) / 1000}s`);
} finally {
  rmSync(target, { recursive: true, force: true });
  execSync("dropdb --if-exists drill_db");
}

Two details make this useful rather than decorative:

Assert a floor, not just success. rows > 0 passes for a database with one test user in it. A threshold that reflects reality is what catches the dump that quietly started emptying.

Fail loudly. A drill that fails silently in CI is worse than no drill, because it manufactures confidence. Wire it to whatever actually wakes someone up.

Run it weekly on a schedule. It costs minutes and it is the only thing in your entire backup setup that tests the thing you care about.

How often

Data	Drill cadence
Primary production database	Weekly, automated
Application data, uploads	Monthly
Full server or cluster rebuild	Quarterly, by hand
Anything after a major schema or infra change	Immediately

That last row is the one people skip. A migration, a version upgrade, a storage change, a new service — every one of them can quietly break the restore path while backups keep reporting success. The drill after a change is worth more than three routine ones.

The full-rebuild drill stays manual on purpose. It is the one that finds the thing that was never in the backup script at all, and finding those requires a human noticing something is missing.

The checklist

✅ Restore into somewhere disposable, never over live data
✅ Load into a real engine, not just onto disk
✅ Compare row counts against production, and read actual rows
✅ Confirm the schema matches what the application expects
✅ Start the application and load a page that reads user data
✅ Time it end to end, and log the number
✅ Assert a floor in automation, not merely "no error"
✅ Tear down the scratch environment when done
✅ Re-drill immediately after any schema or infrastructure change

The point

Backups are the only part of an infrastructure that is never exercised in normal operation. Every other component — the database, the load balancer, the deploy pipeline — is used constantly, so breakage surfaces within hours. A backup pipeline can be broken for a year and look perfect the entire time.

Restore drills are how you exercise it. They are not a compliance box, and they are not about the file; they are the only way the sentence "we have backups" becomes something you actually know instead of something you assume.

Our guides for Postgres, MySQL, MongoDB, Redis, SQLite, S3, DynamoDB, Docker and Kubernetes each end with a drill specific to that system.

Start with the free tier: backupdata.io has 5 GB free, no card, and the 10-minute quickstart gets you to a first verified snapshot today.