Kubernetes backups: what actually needs backing up

Ask three engineers how to back up Kubernetes and you get three answers: snapshot etcd, run Velero, or "it's all in git." Each is right about one third of the problem, and the gap between them is where clusters actually get lost.

There are three separate things in a cluster, they fail independently, and they need different tools:

Backing up etcd protects the first and does nothing for the second. Velero addresses the first two. Git addresses the third — if anyone actually kept it current. This guide covers all three, and is honest about which parts you should not be doing at all.

Start here: do you even have etcd?

kubectl get nodes -l node-role.kubernetes.io/control-plane

If that returns nothing, you are on a managed cluster — EKS, GKE, AKS, DigitalOcean, Civo. The provider runs the control plane, you cannot reach etcd, and you should not try. Their control plane is their problem; skip to section 2, because your risk is entirely in PV data and config.

If you self-host — kubeadm, k3s, RKE, Talos — etcd is yours and section 1 is for you.

1. Cluster state: etcd snapshots

mkdir -p ./k8s-backup

ETCDCTL_API=3 etcdctl snapshot save ./k8s-backup/etcd.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

On k3s, the equivalent is simpler, since k3s manages this itself:

k3s etcd-snapshot save --name manual

Verify it before you trust it:

ETCDCTL_API=3 etcdctl --write-out=table snapshot status ./k8s-backup/etcd.db

That prints hash, revision, total keys and size. A snapshot with a plausible key count is readable; one that errors here would have errored during recovery instead.

Two things worth knowing before you rely on this:

An etcd snapshot contains your secrets in whatever form etcd holds them. Unless encryption at rest is configured on the API server, that is base64, which is not encryption. Treat the snapshot file exactly as you would treat the cluster's root credentials — which is the argument for the client-side encryption in section 4.

Restoring etcd is a whole-cluster operation, not a way to recover one deleted Deployment. It rolls every object back to snapshot time, including objects you wanted to keep. For "someone deleted the wrong thing," manifests are the recovery path, not etcd.

2. Application data: PersistentVolumes

This is the part that actually matters, and the part etcd snapshots do nothing for. etcd stores the PersistentVolumeClaim object — a few hundred bytes describing a request for storage. Your database's actual rows live on a disk somewhere else entirely.

For databases in pods, dump — don't copy files. Same rule as everywhere else: a file-level copy of a live database is a torn copy.

mkdir -p ./k8s-backup/dumps

kubectl exec -n prod deploy/postgres -- \
  pg_dump --format=custom --username=postgres app_db \
  > ./k8s-backup/dumps/app_db.dump

kubectl exec -n prod deploy/redis -- \
  redis-cli --rdb /tmp/dump.rdb && \
  kubectl cp prod/$(kubectl get pod -n prod -l app=redis -o name | cut -d/ -f2):/tmp/dump.rdb \
    ./k8s-backup/dumps/redis.rdb

Our Postgres, MySQL, MongoDB and Redis guides cover the consistency rules for each engine.

For non-database volumes, mount them into a helper pod and tar them out:

# backup-helper.yaml
apiVersion: v1
kind: Pod
metadata:
  name: backup-helper
  namespace: prod
spec:
  restartPolicy: Never
  containers:
    - name: tar
      image: alpine
      command: ["sleep", "3600"]
      volumeMounts:
        - name: data
          mountPath: /data
          readOnly: true
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: my-app-data

kubectl apply -f backup-helper.yaml
kubectl wait --for=condition=Ready pod/backup-helper -n prod

kubectl exec -n prod backup-helper -- tar cf - -C /data . \
  > ./k8s-backup/volumes/my-app-data.tar

kubectl delete pod backup-helper -n prod

Use uncompressed tar. Gzip output changes completely after the first altered byte, so a compressed archive of a barely-changed volume looks entirely new to a deduplicating backup system and re-uploads in full every night. Plain tar keeps unchanged bytes in place. The service compresses on its own.

Note the readOnly: true and that most ReadWriteOnce volumes can only attach to one node — if the helper pod won't schedule, that is why, and it needs to land on the same node as the workload.

What about CSI volume snapshots? If your storage class supports them, VolumeSnapshot objects are fast and consistent. They are also stored by the same storage provider, in the same account, under the same credentials as the volume itself — the same blast radius argument that applies to S3 versioning and DynamoDB PITR. Use them for fast rollback; do not count them as your off-site copy.

3. Config: put it in git, not in a backup

The common advice here is:

kubectl get all --all-namespaces -o yaml > backup.yaml     # don't

This produces a file that cannot be applied back. It is full of status blocks, resourceVersion, uid, creationTimestamp, cluster IPs, and generated names, and kubectl apply rejects or mangles most of it. It also misses everything all does not cover — ConfigMaps, Secrets, Ingresses, PVCs, CRDs, RBAC — which is most of what a cluster actually is.

The honest answer is that manifests belong in version control, not in a backup system. If your cluster's desired state lives in a git repo and is applied from there, config recovery is git clone plus kubectl apply, with full history and review. That is strictly better than any export.

If you are not there yet, capture something applicable in the meantime:

for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
  for kind in deployment statefulset daemonset service ingress configmap pvc cronjob; do
    kubectl get "$kind" -n "$ns" -o yaml 2>/dev/null \
      | grep -v -E '^\s+(resourceVersion|uid|creationTimestamp|selfLink|generation):' \
      > "./k8s-backup/manifests/${ns}-${kind}.yaml"
  done
done

Treat that as a stopgap that buys you a reference copy, not as a restore path. And note that Secrets are deliberately absent from that list — see the encryption note below before you add them.

4. Get it all off the cluster

Everything above lands in ./k8s-backup. A backup that lives in the cluster it protects is not a backup.

Sign in at Backup Data, claim the free 5 GB workspace, mint an API key scoped backup:write, backup:read, snapshots:read, then:

npm install @lighthouse-web3/baas-js-sdk
export LH_API_KEY="lh_xxxxxxxxxxxxxxxxxxxxxxxx"
export LH_WORKSPACE_ID="your-workspace-uuid"

import { BackupClient, generateKeyfile } from "@lighthouse-web3/baas-js-sdk";

const client = new BackupClient({
  apiKey: process.env.LH_API_KEY,
  workspaceId: process.env.LH_WORKSPACE_ID,
});

const snapshot = await client.backup(["./k8s-backup"], {
  description: "nightly cluster backup",
  tags: { type: "kubernetes", cluster: "prod-1" },
  encryption: {
    keyfilePath: "/secure/lh.keyfile",
    passphrase: process.env.LH_KEYFILE_PASSPHRASE,
  },
});

Encryption is not optional here. An etcd snapshot contains every Secret in the cluster — database passwords, API tokens, TLS private keys, registry credentials. Data is AES-GCM encrypted on your machine and the server stores only ciphertext.

You hold the keys. Lose the keyfile and passphrase and the data is unrecoverable, with no server-side reset. And keep the keyfile outside the cluster — a keyfile stored in a Kubernetes Secret is a keyfile that dies with the cluster it was supposed to help you rebuild.

5. Prove it restores

A Kubernetes backup nobody has restored is a strong claim about YAML you have never run.

import { mkdtempSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";

const target = mkdtempSync(join(tmpdir(), "k8s-restore-drill-"));
await client.restore(snapshot.snapshotId, target, {
  onProgress: (e) => console.log(`[${e.phase}] ${e.current}/${e.total}`),
});

Every restore verifies chunk integrity against expected checksums before reassembly.

Drill in a scratch cluster, not production. kind or k3d on a laptop is enough for everything except the etcd path:

kind create cluster --name restore-drill
kubectl create namespace prod

# Config
kubectl apply -f "$TARGET/k8s-backup/manifests/" --namespace prod

# Volume data, into the restored PVC
kubectl apply -f backup-helper.yaml     # without readOnly this time
kubectl exec -i -n prod backup-helper -- tar xf - -C /data \
  < "$TARGET/k8s-backup/volumes/my-app-data.tar"

# Database
kubectl exec -i -n prod deploy/postgres -- \
  pg_restore --username=postgres --dbname=app_db \
  < "$TARGET/k8s-backup/dumps/app_db.dump"

Then check the thing that actually matters: does the application serve traffic? kubectl get pods showing Running is not the test. Port-forward and load a real page.

For etcd specifically, the restore is etcdctl snapshot restore into a new data directory, followed by pointing the control plane at it — a documented but disruptive procedure that stops the API server. Rehearse it on a throwaway self-hosted cluster before you ever need it, because doing it for the first time under pressure is how clusters stay down for hours.

The drill checklist:

• ✅ Restore into a scratch cluster, never the live one
• ✅ Verify etcdctl snapshot status reports a plausible key count
• ✅ Confirm PVC data actually landed, not just that the PVC bound
• ✅ Check file ownership inside the container if the app won't start
• ✅ Port-forward and load a real page — Running is not working
• ✅ Confirm Secrets came back, or that you know where they come from
• ✅ Record wall-clock time — that is your real RTO
• ✅ kind delete cluster when done

What about Velero?

Velero is the incumbent and it is good at what it does: it understands Kubernetes objects natively, orchestrates CSI volume snapshots, and handles namespace-scoped restores without you writing any of the above.

The trade is that it runs inside the cluster it protects, stores backups in a bucket you configure, and adds a CRD-heavy component to maintain. Its snapshots are usually in the same cloud account as the cluster.

They are not mutually exclusive, and the sensible split is: Velero for fast in-cluster recovery, an off-provider copy for the day the account itself is the problem. Nothing in this guide replaces Velero's namespace restore; it covers the case Velero structurally cannot, which is losing the account Velero's bucket lives in.

The pipeline, complete

Most Kubernetes backup setups cover exactly one row and assume it covers the table. The fix is not a bigger tool — it is knowing which row you are actually protecting.

Start with the free tier: backupdata.io has 5 GB free, no card, and the 10-minute quickstart gets you to a first verified snapshot today.