Elasticsearch and OpenSearch backups you can actually restore

Start with the thing that costs people the most time:

You cannot back up Elasticsearch by copying /var/lib/elasticsearch.

Not "it's discouraged" — it does not work. Shards are distributed across nodes, so any single node holds a fraction of each index. Lucene segments are being merged and deleted continuously, so a file-level copy of a running node catches segments mid-merge. And the translog on disk is not guaranteed to be consistent with the segment state you copied. You get a directory that looks complete and produces a cluster that will not start, or worse, starts with silently missing documents.

The snapshot API is the only supported path. It is genuinely good — incremental at the segment level, non-blocking, and safe against a live cluster — but it has three sharp edges that catch nearly everyone.

Everything below applies to OpenSearch too; the API is the same, with _snapshot on both.

1. Register a repository (and the restart nobody mentions)

Snapshots go into a repository, which must exist before you can take one.

If you use a shared filesystem repo, path.repo must be set in elasticsearch.yml on every single node, and that requires a restart of each one. It is a static setting. You cannot set it through the API, and a cluster where only some nodes have it will fail to register the repository with a confusing error.

# elasticsearch.yml — on EVERY node, then rolling restart
path.repo: ["/mnt/es-backups"]

curl -X PUT "localhost:9200/_snapshot/local_repo" -H 'Content-Type: application/json' -d '{
  "type": "fs",
  "settings": { "location": "/mnt/es-backups", "compress": true }
}'

The directory must be a shared mount all nodes can write to — NFS or similar. Each node writes its own shards; a local directory on one node produces an incomplete snapshot.

Or use object storage, which avoids the shared-mount problem entirely:

bin/elasticsearch-plugin install repository-s3
# then add credentials to the keystore, and restart

curl -X PUT "localhost:9200/_snapshot/s3_repo" -H 'Content-Type: application/json' -d '{
  "type": "s3",
  "settings": { "bucket": "my-es-snapshots", "region": "us-east-1" }
}'

Verify before trusting it. This checks every node can actually write:

curl -X POST "localhost:9200/_snapshot/local_repo/_verify"

2. Take the snapshot

curl -X PUT "localhost:9200/_snapshot/local_repo/snap-$(date +%Y%m%d)?wait_for_completion=true" \
  -H 'Content-Type: application/json' -d '{
  "indices": "*",
  "include_global_state": true
}'

include_global_state: true captures cluster settings, index templates, ingest pipelines and legacy templates. Leave it out and you restore your documents into a cluster with none of the configuration that made them useful — no templates, so new indices get default mappings, and no pipelines, so ingestion breaks. It is the second most common omission after the repo restart.

Snapshots are incremental at the segment level: the second snapshot only stores segments not already in the repository. Taking them hourly is far cheaper than it sounds.

For scheduling, prefer the built-in policy over cron — it handles retention too:

curl -X PUT "localhost:9200/_slm/policy/nightly" -H 'Content-Type: application/json' -d '{
  "schedule": "0 30 2 * * ?",
  "name": "<nightly-{now/d}>",
  "repository": "local_repo",
  "config": { "indices": ["*"], "include_global_state": true },
  "retention": { "expire_after": "30d", "min_count": 14, "max_count": 60 }
}'

Check it is running:

curl "localhost:9200/_slm/policy/nightly?human"
curl "localhost:9200/_snapshot/local_repo/_all?verbose=false"

3. Get the repository off the cluster

Here is the gap. A filesystem repository on a mount beside your cluster shares its fate — same rack, same account, same ransomware. An S3 repository lives in the same cloud account as the cluster.

The snapshot API solved consistency. It did not solve independence.

So back up the repository directory itself:

npm install @lighthouse-web3/baas-js-sdk
export LH_API_KEY="lh_xxxxxxxxxxxxxxxxxxxxxxxx"
export LH_WORKSPACE_ID="your-workspace-uuid"

import { BackupClient } from "@lighthouse-web3/baas-js-sdk";

const client = new BackupClient({
  apiKey: process.env.LH_API_KEY,
  workspaceId: process.env.LH_WORKSPACE_ID,
});

const snapshot = await client.backup(["/mnt/es-backups"], {
  description: "nightly elasticsearch repo",
  tags: { type: "elasticsearch", cluster: "prod-1" },
  encryption: {
    keyfilePath: "/secure/lh.keyfile",
    passphrase: process.env.LH_KEYFILE_PASSPHRASE,
  },
});

This works well because a snapshot repository is already content-addressed and append-mostly — most files never change once written, so deduplication means each nightly upload is close to the incremental delta.

Copy the repository with the cluster quiesced, or at least not mid-snapshot. Files are added during a snapshot and the index metadata is written last; catching it halfway gives you a repository referencing segments that are not there yet. Run the copy after the SLM policy completes, not alongside it.

Encrypt — an index frequently contains everything your search box can return, which is usually the whole application's content.

4. Restore, and the third sharp edge

You cannot restore over an open index. Elasticsearch refuses. Close it first, or restore under a different name:

# Option A: close, restore, reopen
curl -X POST "localhost:9200/my-index/_close"
curl -X POST "localhost:9200/_snapshot/local_repo/snap-20260821/_restore?wait_for_completion=true" \
  -H 'Content-Type: application/json' -d '{ "indices": "my-index" }'
curl -X POST "localhost:9200/my-index/_open"

# Option B — better for a drill: restore alongside, under a new name
curl -X POST "localhost:9200/_snapshot/local_repo/snap-20260821/_restore?wait_for_completion=true" \
  -H 'Content-Type: application/json' -d '{
  "indices": "my-index",
  "rename_pattern": "(.+)",
  "rename_replacement": "restored-$1",
  "include_aliases": false
}'

include_aliases: false matters in option B. Restoring aliases alongside a renamed index points your production alias at the restored copy, which is a surprising way to serve stale data to users mid-drill.

Version compatibility is a hard constraint. A snapshot can generally be restored into the same major version or the next one — one major version forward, not two, and never backwards. A snapshot from 7.x restores into 8.x; it does not restore into 9.x, and an 8.x snapshot will not go back into 7.x. This is the thing to check before you plan an upgrade, not after.

Then verify the data, not the green status:

curl "localhost:9200/restored-my-index/_count"
curl "localhost:9200/restored-my-index/_search?size=3&pretty"
curl "localhost:9200/restored-my-index/_mapping?pretty"

A green cluster with an empty index is green. Compare counts against production and read actual documents.

The drill checklist:

• ✅ Restore under a renamed index, never over a live one
• ✅ Set include_aliases: false so production aliases stay put
• ✅ Confirm mappings came back, not just documents
• ✅ Check index templates and pipelines exist (include_global_state)
• ✅ Run a real query your application uses, not just _count
• ✅ Verify the target version is within one major of the snapshot
• ✅ Record wall-clock time — that is your real RTO
• ✅ Delete the restored index when done

Is the index worth backing up at all?

Worth asking honestly. If Elasticsearch is a derived index — everything in it comes from Postgres and a reindex job rebuilds it — then your backup strategy may correctly be "back up Postgres, keep the reindex script working."

The question to answer is: how long does a full reindex take, and can you serve traffic during it? For 50 GB and twenty minutes, skip the snapshots. For 2 TB and eleven hours, that reindex is your RTO, and snapshots are dramatically faster.

And if Elasticsearch holds anything that exists nowhere else — enriched documents, user-generated annotations, ingested logs with no other home — it is a primary datastore regardless of what the architecture diagram calls it, and it needs backing up like one.

The pipeline, complete

Start with the free tier: backupdata.io has 5 GB free, no card, and the 10-minute quickstart gets you to a first verified snapshot today.