Back up model checkpoints before your spot instance disappears

Here's the math that makes checkpoint backups different from every other backup problem: a lost checkpoint has a price tag in GPU-hours.

Forty hours into a fine-tune on 8×A100s at spot prices, your instance gets reclaimed. That's the deal with spot, and it's why it's 60–90% cheaper. If your last checkpoint lives on that instance's disk, you didn't lose a file. You lost a four-figure invoice and two days of wall-clock time, and you get to pay both again.

The failure modes are mundane and constant: spot/preemptible reclamation, CUDA out of memory three lines after a checkpoint save, disk-full on a 200 GB checkpoint directory, a rm -rf in the wrong terminal on a shared dev box, or a rented GPU box (Vast, RunPod, Lambda) that simply won't boot tomorrow. Rented GPUs should be treated as ephemeral by default; anything on their disks is already on borrowed time.

This guide builds the pipeline that makes instance loss a shrug:

1. Checkpoint to disk properly (resumable, atomic)
2. Ship checkpoints off the box: deduplicated, versioned, optionally encrypted
3. Keep retention sane (checkpoints are huge; don't hoard 40 of them)
4. Prove you can resume, the restore drill, training edition

1. Checkpoint like you plan to resume

A checkpoint you can resume from is more than model.state_dict(). Save everything training needs to continue as if nothing happened:

# checkpoint.py
import os, torch

def save_checkpoint(path, model, optimizer, scheduler, epoch, step, extra=None):
    tmp = path + ".tmp"
    torch.save({
        "model": model.state_dict(),
        "optimizer": optimizer.state_dict(),      # skip this and your LR/momentum state resets
        "scheduler": scheduler.state_dict(),
        "epoch": epoch,
        "step": step,
        "torch_rng": torch.get_rng_state(),
        "cuda_rng": torch.cuda.get_rng_state_all(),
        "extra": extra or {},                     # tokenizer hash, config, dataset revision…
    }, tmp)
    os.replace(tmp, path)                         # atomic: never a half-written checkpoint

Two habits that pay off downstream:

• Write to a temp file, then os.replace. A reclamation mid-torch.save otherwise leaves a corrupt file as your latest checkpoint, the worst possible artifact.
• Keep a stable directory layout. For example ./checkpoints/latest.pt overwritten each save, plus ./checkpoints/milestones/epoch-08.pt for keepers, with config.yaml and tokenizer files alongside. Stable paths are what make deduplicated uploads effective.

2. Ship it off the box

The training loop's job is training; don't make it block on uploads. Decouple: training writes checkpoints to disk, and a sidecar step ships the directory after each save (or on an interval).

Set up once: sign in at backupdata.io (free 5 GB workspace), mint an API key scoped backup:write, backup:read, snapshots:read, then on the training box:

npm install @lighthouse-web3/baas-js-sdk
export LH_API_KEY="lh_xxxxxxxxxxxxxxxxxxxxxxxx"
export LH_WORKSPACE_ID="your-workspace-uuid"

The uploader (ship-checkpoint.mjs), tagging snapshots with run and step so the portal reads like an experiment log:

import { BackupClient } from "@lighthouse-web3/baas-js-sdk";

const [runId, step] = process.argv.slice(2);
const client = new BackupClient({
  apiKey: process.env.LH_API_KEY,
  workspaceId: process.env.LH_WORKSPACE_ID,
});

const snapshot = await client.backup(["./checkpoints"], {
  description: `run ${runId} @ step ${step}`,
  tags: { type: "checkpoint", run: runId, step: String(step) },
  hostname: process.env.HOSTNAME ?? "gpu-box",
});

console.log(`snapshot ${snapshot.snapshotId}: ${snapshot.totalSize} bytes covered`);

Call it from the training loop after each save, fire-and-forget so a slow network never stalls a GPU:

import subprocess

def ship_async(run_id: str, step: int):
    subprocess.Popen(
        ["node", "ship-checkpoint.mjs", run_id, str(step)],
        stdout=open(f"ship-{step}.log", "w"), stderr=subprocess.STDOUT,
    )

# in the loop:
if step % checkpoint_every == 0:
    save_checkpoint("./checkpoints/latest.pt", model, opt, sched, epoch, step)
    ship_async(run_id, step)

(No Node on the box? A cron entry running the same script every 15 minutes against the checkpoint directory works fine; the point is that shipping is decoupled from training. A Go binary using the Go SDK is the other zero-runtime option.)

What dedupe does and doesn't buy you here, honestly: uploads are content-addressed and chunked, so bytes identical to already-stored chunks aren't re-uploaded. The static parts of your checkpoint directory (base model you're fine-tuning from, tokenizer, configs, dataset snapshots, frozen layers) dedupe extremely well across runs and saves. Fully-updated weight tensors change densely between steps, so consecutive full checkpoints mostly ship as new (compressed) chunks; LoRA/adapter checkpoints, being small, are trivial either way. You'll see your real ratio in the portal after two uploads. Don't take our word for it.

Proprietary weights? Add client-side encryption. The checkpoint is AES-256-GCM encrypted on your box; the server stores ciphertext only:

const snapshot = await client.backup(["./checkpoints"], {
  description: `run ${runId} @ step ${step}`,
  tags: { type: "checkpoint", run: runId },
  encryption: {
    keyfilePath: "/secure/lh.keyfile",   // generateKeyfile(...) once, then guard it
    passphrase: process.env.LH_KEYFILE_PASSPHRASE,
  },
});

Lose the keyfile and passphrase and the weights are gone. There is no server-side reset. Treat the keyfile like the model IP it protects.

3. Retention: checkpoints are huge, prune like you mean it

Forty retained checkpoints of a 20 GB run is 800 GB of storage for maybe three files you'd ever restore. A policy that matches how people actually resume: keep the recent few plus your milestones.

// keep the 5 newest snapshots; drop anything older than 14 days
const preview = await client.pruneSnapshots({
  keepLatest: 5,
  before: new Date(Date.now() - 14 * 86400_000).toISOString(),
  dryRun: true,                          // always look before you delete
});
console.log("would delete:", preview.snapshotIds);

await client.pruneSnapshots({ keepLatest: 5, before: preview.before, dryRun: false });

A snapshot is deleted only if it's both outside the newest 5 and older than 14 days. Milestone snapshots you want forever: back them up as a separate source (e.g. ./checkpoints/milestones with its own tags) and exclude that source from pruning.

4. The restore drill, training edition

For databases, a restore drill means restoring a dump. For training, it means something stricter: restore the checkpoint on a fresh box and confirm training resumes.

import { mkdtempSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";

const target = mkdtempSync(join(tmpdir(), "lh-resume-drill-"));
await client.restore(snapshotId, target, {
  onProgress: (e) => console.log(`[${e.phase}] ${e.current}/${e.total}`),
});
console.log(`restored into ${target}`);

Then the part that actually proves anything:

ckpt = torch.load(f"{target}/checkpoints/latest.pt", map_location="cpu")
model.load_state_dict(ckpt["model"])
optimizer.load_state_dict(ckpt["optimizer"])
scheduler.load_state_dict(ckpt["scheduler"])
# run 50 training steps and eyeball the loss curve:
# continuous with the pre-checkpoint curve -> real checkpoint.
# a loss spike -> you saved weights but lost optimizer/RNG state. fix it now, not mid-incident.

Restores verify every chunk against its checksum before reassembly, so silent corruption fails loudly at drill time. Time the whole drill (download, load, first resumed step): that's your true recovery cost in GPU-minutes, and it's the number that tells you whether checkpoint_every is set right. Checkpointing every 30 minutes with a 10-minute drill time means an instance reclamation costs you ~40 minutes, worst case. Do the drill once per project, and again whenever the checkpoint format changes.

The pipeline, complete

Spot prices exist because interruption is someone's problem. With checkpoints shipped off the box and a proven resume path, it isn't yours.

Start with the free tier: backupdata.io gives you 5 GB, no card. First checkpoint snapshot in about 10 minutes via the quickstart.