Skip to main content

Back up and restore data

Backups in NKP are opt-in: Longhorn backups stay off until you add a backups.longhorn block with an object-storage target to your cluster config. Once enabled, NKP backs up your cluster's persistent data, including JupyterHub user homes, to off-cluster object storage using Longhorn's native backups. Backups cover volumes on Longhorn, so this guide applies to clusters where Longhorn is the storage layer. It shows you how to enable backups, confirm they are running, and restore data both within a cluster and onto a fresh one. For the full configuration schema (S3-compatible endpoints, Azure, credential and teardown options), see the Longhorn backups reference.

What gets backed up​

When you enable backups, NKP configures two recurring jobs: hourly snapshots and daily backups. These are not the same thing:

  • A snapshot is an in-cluster, point-in-time copy of a volume. It is fast and cheap, but it lives on the same storage as the volume, so it does not survive losing the cluster.
  • A backup copies a snapshot to off-cluster object storage (S3 today; see the Azure note below). This is your disaster-recovery copy.

Every volume on the default longhorn StorageClass is covered automatically, with no per-volume configuration. This includes JupyterHub user homes even when they are idle: their volumes detach when a server is culled, and NKP configures Longhorn to briefly reattach them to take the scheduled backup.

Backups are incremental. After the first full copy, each run uploads only the blocks that changed, so routine backups stay small and fast.

Each backup is self-contained and independently restorable. When retention deletes older backups, Longhorn removes only the data no remaining backup still needs, so pruning never affects the backups you keep.

note

Longhorn backs up whole volumes at the block level, so you cannot exclude individual paths (for example .cache or other regenerable dotfiles) from a home backup. Trimming what gets backed up is a storage-layout decision (keep regenerable data off the backed-up volume), not a backup setting.

Enable backups​

Add a backups.longhorn block to your cluster config and re-run nic deploy. This minimal example enables backups to a new S3 bucket on AWS:

Azure is not yet functional

The schema includes an azure: target, but Azure backups do not work today: NKP does not currently install Longhorn on Azure clusters (their storage layer is managed-csi, not Longhorn), so enabling backups.longhorn on Azure fails nic validate. The azure: block is forward-looking scaffolding for a future release.

backups:
  longhorn:
    enabled: true
    s3:
      bucket: my-nebari-backups
      region: us-west-2
      prefix: my-cluster/
      create_bucket: true # NKP creates the bucket (AWS only today)
      retain_on_destroy: true # default; orphans and keeps the bucket on `nic destroy`
      access_key_id_env: LONGHORN_S3_ACCESS_KEY_ID
      secret_access_key_env: LONGHORN_S3_SECRET_ACCESS_KEY
    schedules:
      snapshot:
        cron: "0 * * * *" # hourly
        retain: 24
        concurrency: 5
      backup:
        cron: "0 3 * * *" # daily at 03:00
        retain: 30
        concurrency: 3

The *_env fields name the environment variables that hold your S3 credentials; export them before you run nic deploy. Set both or neither; nic validate rejects setting only one. On AWS you can instead omit both fields to use keyless EKS Pod Identity auth, where NKP provisions a role-based association for Longhorn and stores no static keys.

For S3-compatible endpoints (MinIO, Wasabi, Cloudflare R2) and the full set of credential and teardown options, see the configuration reference. For a complete, annotated AWS config, including the dedicated Longhorn storage node group this minimal example assumes, see examples/longhorn-backups-config.yaml.

Apply the change:

nic deploy -f <config-file>
Enabling on an existing cluster

Backup resources are applied during the configuration phase of nic deploy, which runs only after the infrastructure phase. On an existing cluster, that phase may roll node groups (for example, to pick up a newer Amazon Machine Image, or AMI), and if the roll fails the deploy aborts before backups are configured; there is no partial apply. If a deploy fails partway, resolve the failure, re-run nic deploy, and confirm backups landed as shown below.

Confirm backups are running​

Check that the backup target is reachable and backups are completing:

kubectl -n longhorn-system get backuptarget default
kubectl -n longhorn-system get backups --sort-by=.metadata.creationTimestamp

A healthy backup target reports AVAILABLE: true. Each backup reaches the Completed state, and the most recent one should fall within your backup schedule (daily by default). You can also browse backups in the Longhorn UI under Backup.

If the target never becomes available, the credentials or bucket settings are wrong. If backups are missing for a volume, confirm it uses the longhorn StorageClass.

Restore a volume in the same cluster​

NKP schedules and configures backups but does not orchestrate restores; restoring is a Longhorn operation.

  1. Stop the workload so its volume detaches. For a JupyterHub home, shut the user's server down from the admin panel or let the idle culler stop it.
  2. Find the Longhorn volume backing the claim: 
    kubectl get pvc <pvc-name> -n <namespace> -o jsonpath='{.spec.volumeName}'
  3. Restore the backup. In the Longhorn UI under Backup, select the volume's backup and choose Restore. Longhorn creates a new volume from the backup.
  4. Bind a claim to the restored volume. On the restored volume in the Longhorn UI, use Create PV/PVC to create a PersistentVolume and a bound claim. To reuse the workload's original claim, delete the old claim first, then set the new claim's name and namespace to match it. For a JupyterHub home, the claim name is what KubeSpawner looks up, so it must match exactly.
  5. Restart the workload, then verify by opening the data in it, not just by checksum. A checksum proves the restore matches the source, not that the source itself was intact. For a JupyterHub home, restart the user's server and KubeSpawner reattaches the claim by name.

For a shared (ReadWriteMany) volume, wait for its share-manager pod in longhorn-system to become Ready before the workload can mount the restored volume.

warning

A restore writes lock objects into the object store, so the credentials must be read-write. A read-only credential fails partway through with failed to put object.

Recover a user's home on a new cluster​

Because the backups live off-cluster, a lost cluster is replaceable: point a new cluster at the same bucket and the backups are still there. For JupyterHub, recovery needs no per-user migration, because a user's home volume name is a pure function of their login. With username_claim: preferred_username and pvc_name_template: claim-{username} (the data science pack defaults), a user's home is always the claim claim-<username>. Restore their backup into a claim of exactly that name and KubeSpawner adopts it on their next spawn.

Teardown deletes backups unless you keep them

This recovery path depends on the bucket surviving the old cluster. retain_on_destroy defaults to true, which orphans the bucket so its contents outlive nic destroy. Setting it to false deletes the bucket (or, on Azure, the storage account and container) along with the cluster: an irreversible loss of every backup, and with it any ability to recover onto a new cluster. See Destroy a cluster.

  1. Deploy the new cluster with the same backups.longhorn.s3.bucket and credentials, and the same JupyterHub username_claim and pvc_name_template as the original.
  2. Recreate the user in the new cluster's Keycloak realm with the same username, so their preferred_username, and therefore their claim name, matches.
  3. Discover the backups. Longhorn syncs the backup target on a poll cycle (about five minutes). To discover them immediately, request a sync: 
    kubectl -n longhorn-system patch backuptarget default --type=merge \
      -p '{"spec":{"syncRequestedAt":"2026-01-01T00:00:00Z"}}'
    Use the current time; any changed value triggers the sync.
  4. Restore the backup into a volume, then use Create PV/PVC on the restored volume to create a claim named claim-<username> in the JupyterHub namespace (jupyterhub by default), following the same-cluster restore steps.
  5. Have the user log in. Their first login creates their JupyterHub record, and starting their server binds the restored claim. Confirm recovery by opening their notebooks in the running session.