What deduplication actually does in a backup system

Short answer: deduplication stores identical data once, however many times it appears. In backups the appearances are mostly the same file backed up on consecutive days, so it turns "30 copies of a 50 GB database" into roughly one copy plus 30 days of differences.

The interesting part is not the idea — it is that there are three ways to implement it and two of them barely work on the data backups actually contain.

Level 1: file-level

Store each unique file once, by hash. Also called single-instance storage.

users.csv     (hash abc123) → stored
users-copy.csv(hash abc123) → pointer to the same blob

Great for a shared drive full of duplicated attachments. Nearly useless for backups, because the thing you back up nightly is one large file whose hash changes every night. One byte differs, and you store the whole 50 GB again.

Level 2: fixed-block

Split every file into fixed 4 KB blocks and hash each one. Now a small change re-stores only the affected blocks.

This works — until anything is inserted rather than overwritten:

Original:  [ABCD][EFGH][IJKL][MNOP]     4 blocks, all known
Insert "X" at the front:
Modified:  [XABC][DEFG][HIJK][LMNO][P]  every block is now different

Adding one byte at the start shifts everything after it across block boundaries. Nothing matches. You store the entire file again.

This is the boundary-shift problem, and it is not a corner case for backups. A pg_dump with a new row near the beginning, a tar archive with a file added, a log file that got a header — all shift the remainder. Fixed-block dedupe quietly stops working on exactly the data you most want it to work on.

Level 3: content-defined chunking

Instead of cutting every 4 KB, decide boundaries from the content itself.

Slide a small window over the data, computing a rolling hash. When the hash matches an agreed pattern — say, the low 13 bits are all zero — declare a chunk boundary there. The condition depends only on the bytes in the window, so:

Original:  [ABCD][EFGH][IJKL][MNOP]
Insert "X" at the front:
Modified:  [XABCD][EFGH][IJKL][MNOP]   only the first chunk changed

The boundary markers are anchored to content, so they move with the data. An insertion perturbs the chunk containing it and the alignment re-synchronises immediately after. Everything downstream still matches.

FastCDC — what this product uses — is an optimised variant of this idea, using a cheaper rolling hash and normalised chunk sizes to get the same resilience with substantially less CPU per megabyte.

This is the difference between deduplication that works on real backup data and deduplication that works in a benchmark.

Where dedupe happens matters too

Source-side (before upload): the client hashes chunks, asks the server which it already has, and uploads only the new ones. Saves bandwidth and storage. Backup windows shrink accordingly.

Target-side (after upload): everything is uploaded and deduplicated on arrival. Saves storage only. You still pay the full transfer every night.

For anything backing up over a network, source-side is the one that changes your life. It is why a 100 GB database with 2% churn finishes in minutes rather than hours.

Scope matters as well. Dedupe within one backup is weakest; across all snapshots is what makes retention affordable; across all machines in a workspace means twenty similar servers store their common content once.

The mistake that silently turns it off

Compressing or encrypting before chunking destroys deduplication.

Both operations are designed to make output look random and unrelated to input. Change one byte of the input and a gzip stream diverges completely from that point on. Chunk the compressed output and no chunk matches yesterday's.

The correct order is: chunk → hash → compress each chunk → encrypt each chunk. Dedupe operates on the plaintext boundaries; compression and encryption apply per chunk afterwards.

Practically, for you:

• Use tar, not tar.gz, when staging directories for backup. The Docker and Linux server guides both call this out.
• Keep output paths stable. Writing dump-2026-08-21.sql instead of overwriting dump.sql still dedupes fine on content, but dated files accumulate locally and confuse retention.
• Do not pre-encrypt files yourself. Let the backup client encrypt after chunking, which is what client-side encryption here does.

What ratio to actually expect

Vendor claims of "20:1 deduplication" are usually measured on the friendliest possible data. Realistic figures for 30 days of daily backups:

That last row is worth internalising. If your dataset is mostly media, deduplication will do very little and you should size storage as though it does not exist.

What it buys you in practice

Deduplication is not primarily a cost saving. It changes what schedules are affordable.

Without it, backing up hourly means 24× the storage and 24× the transfer of a daily. Almost nobody does that, so RPO stays at 24 hours by default.

With source-side content-defined chunking, the second backup of the day uploads only the delta:

# RPO drops from 24h to 1h. Storage and bandwidth barely move.
0 * * * * backup /opt/backup/run.sh >> /var/log/backup.log 2>&1

const snapshot = await client.backup(["./db-dumps"], {
  description: "hourly",
  tags: { type: "db-backup", env: "prod" },
});

Cheaper storage is the visible benefit. A tighter recovery point for free is the one that matters.

Common questions

Does deduplication risk data loss from hash collisions? With a cryptographic hash over a 256-bit space, the probability is far below the probability of undetected disk or memory error. It is not a practical concern.

Does deleting one snapshot break others? Not in a reference-counted system. Chunks still referenced by surviving snapshots are kept — see retention policy.

Does it work with encryption? Yes, when encryption happens after chunking. If a system encrypts the whole file first, dedupe is gone.

Is compression the same thing? No. Compression removes redundancy within one file. Deduplication removes redundancy across files and across time. They stack, in that order.

Start with the free tier: backupdata.io has 5 GB free, no card, and the 10-minute quickstart gets you to a first verified snapshot today.