Skip to content

LogicalBackupElasticsearch

LogicalBackupElasticsearch is one backup of a CamundaCluster that stores its data in Elasticsearch. You create it, or another tool creates it for you.

An orchestration cluster on Elasticsearch holds its data in three places: the web-application indices, the exported Zeebe record indices, and the Zeebe partitions. One LogicalBackupElasticsearch captures all three under one backup ID. The operator takes the backup while the cluster runs. A completed backup is one restore point for a LogicalRestoreElasticsearch.

One resource is one backup. The spec is immutable, and the backup runs once. To take another backup, or to retry a failed one, create a new resource. kubectl get lbes lists the backups with their phase, step, and backup ID.

Before you create a backup, make sure that:

  • The CamundaCluster has spec.backupStorageRef, is not suspended, and publishes status.management.
  • The SecondaryStorageConfig of the cluster carries elasticsearch.snapshotRepository. An ElasticsearchCluster with spec.snapshotStorageRef publishes it. The cluster then shows the name in status.management.backupRepository.
  • The backup lives in the namespace of the cluster.

The backup writes the snapshots of the web-application indices and of the exported Zeebe record indices to the snapshot repository. It writes the Zeebe partition backup to the bucket of spec.backupStorageRef of the cluster. The operator creates no Kubernetes resources from this kind.

The smallest backup names the cluster:

apiVersion: core.camunda.io/v1
kind: LogicalBackupElasticsearch
metadata:
  name: my-cluster-backup
  namespace: my-cluster-ns
spec:
  clusterRef:
    name: my-cluster

graph LR
    LB[LogicalBackupElasticsearch] -.->|clusterRef| CC[CamundaCluster]
    CC -.->|storageRef| SSC[SecondaryStorageConfig]
    CC -.->|backupStorageRef| OSC[ObjectStorageConfig]
    SSC -.->|snapshotRepository| ES["Elasticsearch (external)"]
    LB -->|snapshots| ES
    LB -->|partition backup| BUCKET["Bucket (external)"]
    OSC -.-> BUCKET
Hold "Alt" / "Option" to enable pan & zoom

Exporting

The backup soft pauses exporting before it writes and resumes it at the end. Records still flow into Elasticsearch while exporting is soft paused, and Camunda describes what the mode blocks in Management API. Broker disk usage grows for as long as a backup runs. If a step fails, the backup resumes exporting before it ends as Failed.

One operation at a time

A cluster holds one backup or one restore at a time. A backup that finds another backup or a restore on the cluster waits in Pending with reason BackupInProgress, and the message names the holder. A restore also suspends the cluster, so the backup can wait with reason ClusterSuspended instead. The backup starts on its own when the holder gives the cluster back. A backup that ended as ResumeFailed holds the cluster until you delete it.

Time limits

If the management API or Elasticsearch is unreachable during a step, the backup retries for 10 minutes. After that, the step fails and the backup resumes exporting. The resume of exporting is retried for 30 minutes. After that, the backup ends as Failed with reason ResumeFailed, and exporting stays paused. If the cluster is suspended during the run, the backup waits in its step.

Changes

A backup on a cluster that is still rolling out waits in Pending with reason Progressing before it starts. It also waits while Zeebe runs another Elasticsearch endpoint or snapshot repository than the SecondaryStorageConfig of the cluster declares. The same applies to another bucket than the ObjectStorageConfig declares, and to another Camunda version than status.management.version of the cluster. The backup starts when Zeebe runs what the contracts and the cluster declare.

Do not change the storage or the backup bucket of the cluster while a backup runs. The backup fails, and the message names the recorded and the current value.

Do not change the configuration of Zeebe after the backup started. If Zeebe rolls to another configuration before the operator sees the backup of the Zeebe partitions complete, the step fails at once. The backup then ends as Failed, even if Camunda later reports that part as complete. The message names the recorded and the current config hash.

Do not change the Camunda version of the cluster after the backup started. A version change can keep the config hash of Zeebe. If Zeebe moves to another version before the operator sees the backup of the Zeebe partitions complete, the step fails at once. The message names the version in status.version and the current version. A restore needs the one version that took every part of the backup.

If you delete the cluster during the run, the backup ends as Failed. This is also the case when you create a new cluster under the same name. The backup does not touch the new cluster.

Deletion

When you delete the backup, the operator deletes the snapshots and the partition backup that this backup wrote. If the backup still runs, or ended as ResumeFailed, the operator resumes exporting first. It deletes only snapshots and backups that are its own. If Camunda still reports the backup of the web-application indices as in progress, the deletion waits until it ends. If the cluster publishes no management API, for example while it is suspended, the deletion waits until it does. If the operator can use the management API of the cluster and the pinned ObjectStorageConfig now points to another bucket, the deletion waits too. The operator releases the resource if the cluster or its ObjectStorageConfig is gone, or the cluster was created again. It also releases the resource if it cannot use the published management API and this backup holds no pause of exporting. A missing credentials Secret is an example. In each case, it records the Warning event ArtifactsUnreachable. The event names what it left behind.

Status

Type Reason Meaning What to do
Ready Progressing The backup runs, or it waits for the cluster to finish a rollout or to publish its management API in status.management. Wait. The message names the current step or the cause of the wait.
Ready Completed The backup finished. Ready is True. Nothing. Record status.backupId for a restore.
Ready Failed A step failed. Exporting runs again. Read status.failureMessage. Correct the cause and create a new backup.
Ready ResumeFailed A step failed or finished, and exporting did not resume within 30 minutes. Exporting stays paused. Repair the management API, then delete this backup. The deletion resumes exporting. No other backup of the cluster starts before that.
Ready ClusterSuspended The cluster is suspended: by spec.suspend, by a restore, or by the operator to keep two clusters off one backend. The backup waits. Read the Ready condition of the cluster for the cause.
Ready BackupInProgress Another backup or a restore holds the cluster, or a claim Lease that no backup or restore holds. This one waits. The message names the holder or the claim Lease. Wait. If the message says that the cluster is still paused, delete or repair the named backup. If the message names a claim Lease to delete, delete it.
Ready StorageTypeMismatch The cluster does not store its data in Elasticsearch. Use LogicalBackupRDBMS for a relational cluster.
Ready InvalidReference The cluster, its SecondaryStorageConfig, or its ObjectStorageConfig does not exist, or the cluster publishes no snapshot repository. Read the message. Create the resource, or set snapshotStorageRef on the ElasticsearchCluster.
Ready MissingSecret A credentials Secret does not exist: the one that status.management of the cluster names, or the Elasticsearch credentials. Create the Secret that the message names.
Ready ConnectionFailed The management API or Elasticsearch is unreachable. The step is retried. Make sure that the endpoint answers. After 10 minutes the step fails.

status.phase is Pending, Running, Completed, or Failed. Completed and Failed are terminal. The phase Failed has the reason Failed or ResumeFailed, and ResumeFailed means that exporting also stays paused. status.step is the current step: PauseExporting, BackupHistory, SnapshotRecords, BackupRuntime, or ResumeExporting.

A restore needs these fields:

  • status.backupId keys every part of the backup.
  • status.historySnapshots names the snapshots of the web-application indices.
  • status.repository names the snapshot repository.
  • status.partitionsCount is the partition count of the cluster. A restore must match it.
  • status.version is the Camunda version of the cluster when the backup started. A restore of this backup runs only with the exact same version.
  • status.storageSizes holds the recorded volume sizes of Elasticsearch and Zeebe, when the operator can compute them.

status.history, status.records, and status.runtime report the state of each part: Pending, InProgress, Completed, or Failed. status.workloadConfigHash is the config hash of Zeebe when the backup started. status.failureMessage names the step that failed. status.resumeFailureMessage says why exporting did not resume. status.completionTime is when the backup ended. status.observedGeneration is the last generation that the operator reconciled.

Spec reference

Every field, with its type, whether it is required, and its default:

apiVersion: core.camunda.io/v1
kind: LogicalBackupElasticsearch
metadata:
  name: my-cluster-backup
  namespace: my-cluster-ns
spec:
  # object. Required. The CamundaCluster to back up, in the namespace of this resource. Its secondary storage must be Elasticsearch.
  clusterRef:
    # string. Required. Name of the CamundaCluster.
    name: my-cluster

Validation rules

  • The whole spec is immutable. To retry, create a new resource.
  • spec.clusterRef.name is required and must not be empty. The cluster must live in the namespace of the backup.

A production-shaped example

A backup with a name that carries the date:

apiVersion: core.camunda.io/v1
kind: LogicalBackupElasticsearch
metadata:
  name: my-cluster-backup-20260819
  namespace: my-cluster-ns
  labels:
    backup.my-company.example/schedule: nightly
spec:
  clusterRef:
    name: my-cluster

After the backup completes, kubectl get lbes -n my-cluster-ns shows the phase Completed and the backup ID.