Skip to content

CamundaCluster

A CamundaCluster is one Camunda orchestration cluster: the Zeebe brokers, the gateway, the web applications Operate, Tasklist, and Admin, and optionally the connectors runtime. The operator turns it into StatefulSets, Deployments, and Services for Camunda 8.9 or later, and keeps them healthy.

The cluster owns only its workloads. Secondary storage comes from a SecondaryStorageConfig, and bucket storage from an ObjectStorageConfig. Both are contracts: resources that publish the address and the credentials of a storage for other resources to reference. Shared settings come from a CamundaPlatformConfig, sizing defaults from a CamundaClusterPreset, and the versions from a CamundaRelease. Backups attach from the outside through LogicalBackupElasticsearch and LogicalBackupRDBMS.

The smallest cluster names a platform configuration, a version, and a storage contract:

apiVersion: core.camunda.io/v1
kind: CamundaCluster
metadata:
  name: my-cluster
  namespace: my-cluster-ns
spec:
  platformConfigRef: "my-platform-config"
  version: "8.9.0"
  storageRef: "my-storage-config"

graph LR
    CC[CamundaCluster] -.->|platformConfigRef| PFC[CamundaPlatformConfig]
    CC -.->|presetRef| CCP[CamundaClusterPreset]
    CC -.->|releaseRef| CR[CamundaRelease]
    CC -.->|storageRef| SSC[SecondaryStorageConfig]
    CC -.->|backupStorageRef / documentStorageRef| OSC[ObjectStorageConfig]
    CC --> WL["StatefulSet, Deployments, Services"]
    LBE[LogicalBackupElasticsearch] -.->|clusterRef| CC
    LBR[LogicalBackupRDBMS] -.->|clusterRef| CC
Hold "Alt" / "Option" to enable pan & zoom

Topology

Zeebe, the gateway, and the web applications run from one Camunda image. Each block of the spec says where its process runs:

  • zeebe is always a StatefulSet of brokers with persistent volumes.
  • gateway runs Standalone (its own Deployment) or Embedded (inside the brokers). The default is Standalone.
  • operate, tasklist, and admin each run Standalone (their own Deployment) or Embedded. The default is Embedded. An embedded web application runs inside the gateway when the gateway is standalone, otherwise inside the brokers.
  • connectors is a separate runtime. When enabled, it is always its own Deployment.

A cluster that gives Operate its own pods and runs connectors:

apiVersion: core.camunda.io/v1
kind: CamundaCluster
metadata:
  name: my-cluster
  namespace: my-cluster-ns
spec:
  operate:
    mode: Standalone
    replicas: 2
  connectors:
    enabled: true
    version: "8.9.7"
  # ... the rest of your cluster

Endpoints

Each enabled process gets a workload and a Service named <name>-<component>. A standalone gateway serves on the Service <name>-gateway, with gRPC on port 26500 and HTTP on port 8080. An embedded gateway serves the same ports on <name>-zeebe. HTTP serves the REST API under /v2/ and the embedded web applications under /operate/, /tasklist/, and /admin/. A standalone web application serves on its own Service on port 8080.

With an embedded gateway, the operator also creates the Service <name>-zeebe-client. It carries only the ports 26500 and 8080, and it sends traffic only to brokers that are ready. Route traffic from outside the cluster to this Service, not to <name>-zeebe. status.endpoints names the Service for each path. See Routes.

status.gateway publishes the two client addresses, so you do not have to work out which Service runs the gateway. See Status.

Each workload, its pods, and its Service carry the labels camunda.io/cluster and camunda.io/component. The component is one of zeebe, gateway, operate, tasklist, admin, and connectors. Discovery labels lists the labels that you can rely on. List the workloads of a cluster:

kubectl get deploy,sts,svc -n my-cluster-ns -l camunda.io/cluster=my-cluster

A Service name has at most 63 characters. When the cluster name is too long for its suffix, the operator cuts the name and adds a hash of the full name. The camunda.io/cluster label and the derived Secret names carry the same cut form. For such a cluster, read the label value with kubectl get deploy --show-labels and select on that value.

The operator creates no Ingress. You route traffic to the cluster. spec.externalUrl tells the cluster its public base URL for OIDC redirects and links.

Authentication

The authentication method, basic or OIDC, comes from the CamundaPlatformConfig that platformConfigRef names. A cluster cannot choose its own method.

Under basic authentication the operator creates the admin user admin. It stores the credentials in the Secret <name>-camunda-admin, under the keys username, password, and email. To rotate the password, set spec.auth.basic.passwordRotation to a new value. status.adminPassword.rotation records the value that the operator applied.

Under OIDC the identity provider authenticates every caller, and spec.auth.admin names the identities that get the admin role. An OIDC cluster without spec.auth.admin has no administrator.

The authentication guide explains both methods, the admin role, and the rotation with its failure modes.

Version

The effective version is spec.version, or the version of the CamundaRelease of releaseRef when the field is absent. A release can also pin the image to pull, but the rules below read the version, not the image. spec.connectors.version uses the same minor as the Camunda version, but its own patch. Set it on its own.

Upgrade

To upgrade, raise spec.version, or raise the version of the release that releaseRef names. The operator rolls every workload to the new version, except Connectors, which follows spec.connectors.version. When you move to a new minor, raise spec.connectors.version to that minor too. kubectl get camundacluster shows the new version at once, and Ready reads Updating until every process is healthy again.

Move one minor at a time, and move to the latest patch of your minor first. Camunda blocks the start after a skipped minor, see Version compatibility checks. The operator refuses a version that skips a minor before it rolls a pod, see A skipped minor is refused. Prepare for upgrade lists what changes in each minor. An attached CamundaOptimize reports VersionMismatch until you raise its version to the same minor.

Wait between two minor upgrades in a row. Each broker must take a snapshot on the new minor before you move again. By default, a broker takes a snapshot every five minutes. A broker that finds a snapshot from two minors back refuses to start, see Upgrade Zeebe. The operator does not refuse this move. It compares the new version with the version that its brokers run, not with their snapshots.

A skipped minor is refused

The operator refuses a version more than one minor above the version that the brokers run. The cluster reports Ready: False with reason VersionSkipRefused and records a Warning event with the same name. It applies no change of the spec, and the brokers keep the version they have.

status:
  observedGeneration: 5
  conditions:
    - type: Ready
      status: "False"
      reason: VersionSkipRefused
      message: 'the effective version 8.11.0 skips a minor: the brokers run 8.9.9.
        Camunda upgrades one minor at a time, and a broker that starts after a skipped
        minor marks itself unhealthy and applies no migration. Set the version to
        the latest patch of 8.10 first, and move to 8.11.0 once every broker runs it'
      observedGeneration: 5
      lastTransitionTime: "2026-10-11T09:14:00Z"

To recover, set the version to the latest patch of the next minor. When Ready is True again, every broker runs it. Then wait for a snapshot, as Upgrade says, and set the version that you want. No annotation lets a skipped minor through. camunda.io/allow-version-downgrade applies to a lower version only.

The rule reads the effective version and the running version, as the downgrade rule below does. A raised spec.version, a removed spec.version over a newer release, and a raised release all meet it. A cluster that you create again on retained volumes obeys the rule too. A change of major is not a skipped minor, and the operator does not refuse it. Camunda does not support a change of major, see Version compatibility checks.

A suspended cluster reports the refusal when it resumes. It then stays at zero until you set a version that the brokers can move to.

A lower version is refused

The operator refuses a version below the one that the brokers run. Camunda does not support a downgrade of a running cluster, see Version compatibility checks. The cluster reports Ready: False with reason VersionDowngradeRefused and records a Warning event with the same name. It applies no change of the spec, and the brokers keep the version they have.

status:
  observedGeneration: 4
  conditions:
    - type: Ready
      status: "False"
      reason: VersionDowngradeRefused
      message: 'the effective version 8.9.0 is below the running version 8.9.5. Camunda
        does not support a downgrade of a running cluster: a broker that starts on
        data that a newer version wrote marks itself unhealthy. Set the version to
        8.9.5 or later. To run 8.9.0 on the data of a backup taken with it, restore
        that backup after that: the restore sets the version itself, and it cannot
        start while this refusal stands. To downgrade on purpose over the data the
        brokers have, set the annotation camunda.io/allow-version-downgrade="8.9.0"
        on the cluster'
      observedGeneration: 4
      lastTransitionTime: "2026-08-20T09:14:00Z"

The rule reads the effective version. A lower spec.version, a removed spec.version over a lower release, and a lowered release all meet it.

The running version is the version that the operator last applied to the brokers, even before the pods have rolled. The operator also writes it on each broker volume, as the annotation camunda.io/broker-version. So a cluster that you create again on retained volumes (Storage) obeys the rule too. A new cluster with new volumes has no running version.

A suspended cluster reports the refusal when it resumes. It then stays at zero until you set the version forward again or sanction the downgrade.

Downgrade on purpose

CAUTION: Do not downgrade a cluster over data that a newer version wrote. To recover, set the version back to the one that wrote the data. To run the lower version on the data of a backup taken with it, restore that backup. Do not lower spec.version for it. The restore sets the version itself, see LogicalRestoreElasticsearch and LogicalRestoreRDBMS. A restore cannot start while the refusal stands.

To downgrade on purpose, set the annotation camunda.io/allow-version-downgrade to the target version. Set spec.version to the same value in the same edit.

apiVersion: core.camunda.io/v1
kind: CamundaCluster
metadata:
  name: my-cluster
  namespace: my-cluster-ns
  annotations:
    camunda.io/allow-version-downgrade: "8.9.0"
spec:
  version: "8.9.0"
  # ... the rest of your cluster

The annotation sanctions a move to that version and to no other. The operator removes the annotation when the brokers carry the version. It also removes an annotation that does not name the effective version. You can also set the annotation after the refusal, because the refusal keeps the lower version pending.

After a restore, you can give the release control of the version again. Set the annotation to the version of the release, and remove spec.version in the same edit.

Storage

The brokers keep their data on one PersistentVolumeClaim per pod. When spec.zeebe.storageSize grows, the operator expands every bound broker volume in place, without a restart. The storage class must allow volume expansion. The operator never shrinks a volume. The API server rejects a smaller inline value that replaces an inline value. A smaller size from a preset, or a first inline value below the current size, is ignored, and the cluster records the Warning event StorageShrinkIgnored.

The storage class of the broker volumes cannot change after the operator created the broker StatefulSet <name>-zeebe. The API server rejects a change of spec.zeebe.storageClassName after you set it. A new class can still come from the preset, or from a class that you add to a cluster that inherited one. The operator then keeps the class of the StatefulSet, also while the cluster is suspended. It records the Warning event StorageClassChangeIgnored when it ignores a new requested class, also when the preset removes its class. A cluster without a broker StatefulSet takes the new class. Volumes that whenDeleted: Retain kept from a deleted cluster of the same name keep their class.

spec.zeebe.persistentVolumeClaimRetentionPolicy.whenDeleted decides what happens to the volumes when you delete the cluster. Delete (the default) removes them. Retain keeps them for a later cluster with the same name. A scale-down and a suspension always keep them.

Secondary storage

spec.storageRef names the SecondaryStorageConfig in the namespace of the cluster. This contract tells the cluster where its backend is: the Elasticsearch or the database that holds its secondary storage. The secondary storage guide shows how to set up Elasticsearch or PostgreSQL.

One CamundaCluster writes one backend. Camunda fixes the index names and the tables, so two clusters on one backend write each other's data. A restore of one of them deletes the data of the other.

The operator knows a backend by the address that the contract resolves to, not by the contract:

  • For Elasticsearch, the address is the scheme, the host, and the port of the endpoint. A path prefix does not make a second backend.
  • For an RDBMS, the address is the host, the port, and the database name.
  • Two contracts that resolve to one address are one backend, in any namespace.
  • A bare Service name, such as es, resolves in the namespace of the contract. So es in two namespaces is two backends, and es beside es.<namespace>.svc is one.

The operator compares addresses, not servers. Two contracts that reach one Elasticsearch through two host names are not caught. Give one backend one address.

The first cluster on a backend holds it until the cluster moves to another backend or you delete the cluster. A cluster whose contract you delete keeps its backend and keeps running.

A second cluster on a held backend

The API server accepts a second cluster on a held backend. The operator suspends that cluster: every workload at zero, and the volumes kept. Its Ready is False with reason StorageAlreadyAttached, and the message names the holder and the backend.

status:
  conditions:
    - type: Ready
      status: "False"
      reason: StorageAlreadyAttached
      message: >-
        CamundaCluster "my-cluster-ns/my-other-cluster" already holds the
        backend "elasticsearch|https://es-http.my-cluster-ns.svc:9200". One
        CamundaCluster holds one backend, so this cluster stays suspended
        until that cluster moves to another backend or is deleted

When you delete the holder, or move it to another backend, the waiting cluster takes the backend and starts on its own. A paused holder keeps the backend until you unpause it. Two clusters that swap backends in one edit both start. A holder that you point away and back again can find the backend taken, and then reports StorageAlreadyAttached itself.

Waiting for a handover

A cluster does not start while something else still writes its backend. It reports Ready: False with reason WaitingForHandover, keeps every workload at zero, and keeps the volumes. The state clears on its own. The cluster waits for these writers:

  • Pods of another cluster, or of its CamundaOptimize, on the backend, in any namespace. When you delete that cluster, its pods stop after the cluster is gone. When you move that cluster to another backend, its old pods stop as its rollout replaces them.
  • Workloads of another cluster that can still start such a pod: a StatefulSet, a Deployment, a ReplicaSet that asks for replicas, or a Job.
  • A restore into another cluster, see Restores hold the backend.

A running cluster that you move to such a backend stops the same way. The message names the backend and what still writes it:

status:
  conditions:
    - type: Ready
      status: "False"
      reason: WaitingForHandover
      message: >-
        Pods of another cluster, or of its Optimize instance, or workloads
        that start one, still write the backend
        "elasticsearch|https://es-http.my-cluster-ns.svc:9200":
        my-cluster-ns/my-other-cluster-zeebe-0. This cluster starts when they
        are gone

If the named pods never go, delete them. If a named workload keeps them coming back, scale it to zero or delete it.

The pods that the operator runs on a backend carry the label camunda.io/storage-claim, whose value names the backend. Find every pod on one backend by that value:

kubectl get pods -n my-cluster-ns -L camunda.io/storage-claim
kubectl get pods -A -l camunda.io/storage-claim=camunda-storage-8bd62d6c1f48cf988b142a51c9e7010d105e168c

Restores hold the backend

A restore into another cluster holds the backend while it runs. The next cluster on the backend waits with WaitingForHandover, and the message names the restore, for example LogicalRestoreElasticsearch my-cluster-ns/my-other-cluster-restore.

The hold lasts until the restore reaches Completed or Failed, even when the restore stops making progress. It also lasts when you delete the target of the restore, or point it at another backend. A LogicalRestoreRDBMS or a PointInTimeRestore holds the database by its DatabaseServerConfig and database name. The hold stays when that contract moves to another host or port. A LogicalRestoreElasticsearch holds Elasticsearch by the SecondaryStorageConfig of its target. That hold stays when you move the endpoint of that SecondaryStorageConfig. A restore into this cluster itself is no reason to wait. To free the backend from a restore that does not move, delete the restore. A deleted restore keeps the backend until its work stops. Each restore page says when its work stops:

A restore that is gone still holds the backend

This happens only when somebody removed the finalizers of a restore by hand. The restore is gone, but its hold stays, and nothing removes the hold for you. The hold is a Lease in the namespace of the operator, camunda-operator-system in the commands below.

Before you remove it, make sure that the work of the restore has stopped. For a LogicalRestoreRDBMS, no pod with its storage claim label runs. For a LogicalRestoreElasticsearch, no index recovery is active. For a PointInTimeRestore, spec.pitr.lastRecovery on the DatabaseServerConfig answers its request.

  1. List the Leases. The WRITER column shows the restore that the WaitingForHandover message names:

    kubectl get leases -n camunda-operator-system \
  -l app.kubernetes.io/managed-by=camunda-operator,camunda.io/component=storage-writer \
  -o custom-columns='WRITER:.metadata.annotations.camunda\.io/storage-writer,UID:.metadata.labels.camunda\.io/writer-uid'

  2. Delete every Lease of that restore, by the value of its UID column:

    kubectl delete leases -n camunda-operator-system \
  -l app.kubernetes.io/managed-by=camunda-operator,camunda.io/component=storage-writer,camunda.io/writer-uid=0d9c2a41-5e0b-4c7f-9a53-2f1e8b6c7d10

The waiting cluster starts a short time after the last Lease is gone.

Secondary storage over TLS

When the SecondaryStorageConfig names a certificate authority under elasticsearch.caSecretRef, the brokers, the gateway, and the web applications trust that authority. A cluster on an ElasticsearchCluster gets this without a step from you, because that kind fills caSecretRef itself. For an Elasticsearch of your own behind a private authority, set caSecretRef on the contract.

The Zeebe Elasticsearch exporter needs this trust. It has no TLS setting of its own (camunda/camunda#9839). Without caSecretRef it writes no records, and CamundaOptimize stays empty. Every TLS client in those processes then trusts the authority, not only the exporter.

The trust arrives through JAVA_TOOL_OPTIONS. If you set that variable yourself, read Environment and JVM.

Index replicas

spec.indexReplicas sets the number of replicas of each index that the cluster creates in an Elasticsearch secondary storage.

apiVersion: core.camunda.io/v1
kind: CamundaCluster
metadata:
  name: my-cluster
  namespace: my-cluster-ns
spec:
  indexReplicas: 1
  # ... the rest of your cluster

When you do not set it, the node count of the SecondaryStorageConfig gives the count. One node gives 0 replicas. Two or more nodes give 1 replica. The contract of an ElasticsearchCluster always carries the node count. A contract without a node count leaves the count to Camunda.

Elasticsearch never puts a replica on the node that holds its primary. An index with more replicas than the other nodes can hold stays at yellow health, and the ElasticsearchCluster is then not Ready. If you set a count that the nodes cannot place, the cluster runs with it and records the Warning event IndexReplicasExceedNodes.

The cluster applies the count to its existing indices each time it starts. A change of indexReplicas restarts the cluster, and so does a change of the storage contract. A relational secondary storage ignores the field.

If your extraEnv or extraEnvFrom sets CAMUNDA_DATABASE_INDEX_NUMBEROFREPLICAS or ZEEBE_BROKER_EXPORTERS_CAMUNDAEXPORTER_ARGS_INDEX_NUMBEROFREPLICAS, give indexReplicas the same value. Camunda does not start when they differ.

Backups

Without spec.backupStorageRef the cluster takes no backups. With it, the brokers write primary-storage backups to the referenced bucket. On S3 and GCS, each cluster writes under its own prefix, so clusters can share a bucket. On Azure Blob, give each cluster its own container. A second cluster on the same Azure contract reports InvalidReference.

On the Elasticsearch path the storage contract must carry elasticsearch.snapshotRepository, or the cluster reports InvalidReference. On the RDBMS path spec.backup.primaryStorage configures the backup scheduler of Zeebe. The API reference lists its defaults. With a schedule of none, continuous defaults to false.

On the RDBMS path, the backup credentials of the DatabaseConfig feed the dump Job. When they do not resolve, the cluster keeps running and records the Warning event DumpCredentialsUnresolved. The backups of the cluster then wait. The backup guide covers the backup kinds.

The dump Job runs the image postgres:<major>, with the PostgreSQL major of the database server. Set spec.backup.dump.postgresImage to use another image, for example in an air-gapped installation.

Workload identity

The pods of the cluster, except the connectors, run under the default ServiceAccount of the namespace, unless the cluster needs an account of its own. The cluster gets one when spec.serviceAccount is set, or when a referenced bucket has no credentials Secret and authenticates through workload identity. That account is <name>-camunda, or spec.serviceAccount.name. status.serviceAccountName shows the account that the pods use.

While neither the cluster nor its preset sets spec.serviceAccount and no referenced bucket uses workload identity, the cluster needs no account. The operator then deletes <name>-camunda only when the cluster owns it, because the operator created it or took it over. A ServiceAccount <name>-camunda that you created stays.

When the cluster needs <name>-camunda and you created a ServiceAccount of that name, the operator takes it over. It adds its owner reference and its annotations. From then on, the operator deletes the account when the cluster no longer needs it, and when you delete the cluster. To keep an account of your own, give it another name, or set create: false.

When a referenced bucket names a workload identity, the operator puts the matching annotation on the ServiceAccount. ObjectStorageConfig lists the annotations. An annotation in spec.serviceAccount.annotations wins over the derived one. A bucket without an identity block adds no annotation, and you bind the principal system:serviceaccount:<namespace>:<name>-camunda on the cloud side. Two buckets that name different identities of one cloud report InvalidReference.

spec.documentStorageRef names a bucket for document storage. The operator uses only the workload identity of that bucket, as for the backup bucket. It does not configure the document store of Camunda.

The cloud reads the annotations of the ServiceAccount when a pod starts. So when the annotations that the operator puts on the ServiceAccount change, the pods that run under it restart. These are the brokers, the gateway, and the web applications. This holds for an edit of spec.serviceAccount.annotations and for a referenced bucket that names another identity. These changes do not restart the connectors. Any other change to a referenced bucket restarts the same pods, see Changes and referenced Secrets. A backup Job that starts after the change runs with the new annotations.

With serviceAccount.create: false, the operator does not create or change the ServiceAccount, and it does not read its annotations. It reports InvalidReference while the ServiceAccount does not exist. A change to the annotations of that account restarts no pod. After such a change, restart the pods yourself:

kubectl rollout restart statefulset/my-cluster-zeebe -n my-cluster-ns
kubectl rollout restart deployment -n my-cluster-ns \
  -l 'camunda.io/cluster=my-cluster,camunda.io/component in (gateway,operate,tasklist,admin)'

The second command restarts the gateway and each web application that runs in a Deployment of its own. When all of them run in the brokers, it finds no Deployment, and the first command is enough.

When you switch create from true to false, the operator removes its owner reference from the ServiceAccount that it created. That is its last change to the account. The labels and annotations stay, and the account stays when you delete the cluster. The operator no longer sets its labels or annotations, so you can change them. When you switch back to create: true, the operator owns the account again and sets the annotations of the spec.

Connectors

The connectors never run under the ServiceAccount of the cluster, and the identities of the buckets never reach them. Process authors model the calls that the connectors make, so with the role of the cluster a process can read or delete backups. The connectors read documents through the REST API of the cluster, and Camunda says not to give them the document store role (IAM Roles for Service Accounts).

When neither the cluster nor its preset sets connectors.serviceAccount, the connectors run under the ServiceAccount default of the namespace. To give them a cloud role of their own, set the block:

apiVersion: core.camunda.io/v1
kind: CamundaCluster
metadata:
  name: my-cluster
  namespace: my-cluster-ns
spec:
  connectors:
    enabled: true
    version: "8.9.7"
    serviceAccount:
      annotations:
        eks.amazonaws.com/role-arn: "arn:aws:iam::123456789012:role/my-cluster-connectors"
  # ... the rest of your cluster

The connectors then run under my-cluster-connectors, or under spec.connectors.serviceAccount.name. That account carries only the annotations of the block. A change of these annotations restarts the connectors pods, and no other pod. For an identity without an annotation, bind the principal system:serviceaccount:<namespace>:<name>-connectors on the cloud side. status.connectorsServiceAccountName shows the account. When the block is gone from the cluster and from its preset, the connectors go back to the ServiceAccount default. The account that the operator created stays until you delete the cluster.

While connectors are not enabled, the operator creates no connectors account. When connectors are enabled and create is false, Ready is False with reason InvalidReference while the account does not exist. When you create the account, the operator finds it by itself, without an edit to the cluster. After a change to the annotations of that account, restart the connectors pods yourself, with kubectl rollout restart deployment/my-cluster-connectors -n my-cluster-ns. When you switch spec.connectors.serviceAccount.create to false, the operator removes its owner reference from the connectors account, as for the cluster account. When you switch back to true, the operator owns the account again.

The connectors account must not have the name of the cluster account. When spec.serviceAccount.name is default, enabled connectors need spec.connectors.serviceAccount, or Ready is False with reason InvalidReference. The API server rejects a spec in which spec.connectors.serviceAccount.name equals spec.serviceAccount.name. When only one of the two names is set and it equals the derived name of the other account, Ready is False with reason InvalidReference. These two InvalidReference causes apply only while connectors are enabled.

Azure Workload ID

On Azure, a pod needs the label azure.workload.identity/use: "true" as well as the annotation azure.workload.identity/client-id on its account. When spec.serviceAccount.annotations carry the client ID, the operator adds the label to every pod that runs under the cluster account, the backup Jobs included. When spec.connectors.serviceAccount.annotations carry it, the operator adds the label to the connectors pods only.

The operator reads the annotations of the spec, not of the account. With create: false, the operator does not read the annotations of the existing account. If that account carries the client ID, add the label azure.workload.identity/use: "true" through podLabels or connectors.podLabels yourself.

Environment and JVM

The operator renders its own environment first. Then it adds your entries in this order:

  1. The top-level extraEnv.
  2. The extraEnv of the embedded gateway (on the brokers).
  3. The extraEnv of every embedded web application that the process hosts.
  4. The extraEnv of the process itself.

A later entry with the same name wins, and your entry replaces an operator entry with the same name. extraEnvFrom sources are added in the same order. Connectors get the top-level entries and their own block only.

When two tools apply the cluster, for example your GitOps tool and the operator, each extraEnv list merges by name. Each tool owns the entries that it applies, and another tool can add entries next to them. An entry sets value or valueFrom, never both. A CamundaManagementCluster that serves the cluster owns four names, see Management plane. extraEnvFrom and spec.backup.dump.extraEnv do not merge. A tool that applies one of these lists replaces the whole list.

Every Camunda process gets JAVA_TOOL_OPTIONS=-XX:+ExitOnOutOfMemoryError, so the kubelet restarts a pod after an OutOfMemoryError. The heap size comes from the container-aware defaults of the JVM. To change the JVM options, set JAVA_TOOL_OPTIONS in the extraEnv of the process. Keep -XX:+ExitOnOutOfMemoryError in your value. When the storage contract names a certificate authority, the operator adds the trust store options after your value.

To use a trust store of your own, name it with -Djavax.net.ssl.trustStore in your value. The operator then adds no trust store options, and the JVM reads your store only. That store must hold the certificate authority of the Elasticsearch endpoint, or the exporter fails and Optimize reads no records. This is also how you trust a second private authority, for example an OIDC provider or a backup store. Put every authority in one store. The spec has no volume field, so build the Camunda image with the store in it. Then name the image under images.camunda on the CamundaPlatformConfig, or pin it on a CamundaRelease.

A JAVA_TOOL_OPTIONS entry that reads its value from a Secret or a ConfigMap cannot take the trust store options. The cluster records the Warning event TrustStoreOptionsNotApplied, which names the processes. The operator still builds the store at /etc/camunda/es-truststore/cacerts with the password changeit. Name that store in the referenced value, or name a store of your own that holds the authority.

Management plane

A CamundaManagementCluster can serve this cluster. It selects clusters in every namespace through a label selector, so nothing on this resource points at it. A cluster that it serves carries the annotation camunda.io/management-cluster:

apiVersion: core.camunda.io/v1
kind: CamundaCluster
metadata:
  name: my-cluster
  namespace: my-cluster-ns
  annotations:
    camunda.io/management-cluster: my-management-ns/my-management

While the management plane sets spec.console, it also adds four entries to spec.extraEnv, which make the cluster report to Console: CAMUNDA_CONSOLE_PING_ENABLED, CAMUNDA_CONSOLE_PING_ENDPOINT, CAMUNDA_CONSOLE_PING_CLUSTERNAME, and CAMUNDA_CONSOLE_PING_PINGPERIOD. On Camunda 8.10 and later the four names are CAMUNDA_HUB_PING_*. The management plane owns these names and replaces what you set under one of them. To keep an entry of your own, give it another name. The entries change spec, so the pods roll once when they arrive and once when they go.

To remove the annotation and the entries, take the cluster out of spec.clusterSelector of the CamundaManagementCluster. Change the selector, or remove the label that it matches on. Removing spec.console from the management plane removes the four entries only. If you delete the annotation or the entries by hand, the management plane writes them again.

kubectl get camundamanagementcluster -A shows the management planes. status.clusters of each one lists the clusters that it serves.

Monitoring

When spec.monitoring.serviceMonitor.enabled is true, the operator creates one ServiceMonitor per process, named like its workload. It scrapes /actuator/prometheus on port 9600 of a Camunda process and on port 8080 of connectors. On a Kubernetes cluster without the ServiceMonitor kind, the operator creates none and reports no error.

apiVersion: core.camunda.io/v1
kind: CamundaCluster
metadata:
  name: my-cluster
  namespace: my-cluster-ns
spec:
  monitoring:
    serviceMonitor:
      enabled: true
      labels:
        prometheus: "platform"
  # ... the rest of your cluster

Changes and referenced Secrets

A change to the cluster or to a referenced resource rolls out to the pods on its own. So does a change to a key that the cluster reads from a referenced Secret. A change to the labels or annotations of a Secret, or to a key that the cluster does not read, rolls nothing. The CamundaPlatformConfig is cluster-scoped, so the operator copies the Secrets it names in another namespace into the namespace of the cluster. Each copy follows its source.

A change to a referenced ObjectStorageConfig or its credentials Secret restarts the brokers, the gateway, and every web application, also one in its own Deployment. The connectors keep running, because they do not use the buckets.

The API server accepts a cluster that names something you did not create yet, so you can create the resources in any order. A missing CamundaPlatformConfig, CamundaClusterPreset, CamundaRelease, SecondaryStorageConfig, DatabaseConfig, DatabaseServerConfig, or ObjectStorageConfig sets Ready to False with reason InvalidReference. A missing Secret or key sets reason MissingSecret.

When one of these checks fails for a running cluster, the workloads stay up on the configuration that the operator applied last. Ready carries the failure, and the cluster keeps its backend. When the check passes again, the cluster takes the change. A suspension still stops the workloads while a check fails, see Suspend and pause.

status:
  conditions:
    - type: Ready
      status: "False"
      reason: MissingSecret
      message: Secret my-cluster-ns/my-storage-credentials not found
    - type: ZeebeReady
      status: "True"
      reason: Healthy

Suspend and pause

spec.suspend: true scales every workload to zero and keeps the broker volumes. Ready is True with reason Suspended, and status.management and status.gateway are empty. status.endpoints stays, see Routes. When you set suspend back to false, Ready reads Updating until the workloads are healthy again. A backup of a suspended cluster waits with reason ClusterSuspended.

suspend also stops the workloads while a reference check fails. You do not have to correct the reference first. Ready then reports the failure, and the message says why the workloads are at zero:

status:
  conditions:
    - type: Ready
      status: "False"
      reason: MissingSecret
      message: >-
        Secret my-cluster-ns/my-storage-credentials not found. The workloads
        are scaled to zero because the cluster is suspended

When the suspension ends while the check still fails, the stopped workloads stay at zero until the check passes. The message of Ready then ends with The workloads that stopped stay at zero until the reference check passes.

If the API server refuses to stop a workload, that workload keeps running and its endpoints stay published. The cluster records the Warning event WorkloadStopRefused, which names the workload and carries the refusal.

Every suspension reaches the CamundaOptimize attached to this cluster. Its webapp and its importer scale to zero with the cluster, and they start again when the cluster resumes.

spec.pause: true freezes the cluster. The operator changes nothing that it manages for this cluster, and it writes no status. It records a Paused event each time it looks at the resource. Set pause back to false, and the operator continues.

Why the cluster is at zero replicas

The operator also holds a cluster at zero on its own. spec.suspend stays yours. The reason on Ready tells you who holds it. Ready: True means that you asked for zero. Ready: False means that a step of yours is needed, or that something else must finish first.

Ready Reason Cause What to do
True Suspended You set spec.suspend: true. Set suspend: false to resume.
False SuspensionHeld A restore writes the storage of this cluster. Wait for the restore. For a failed restore, see Suspension holds.
False StorageAlreadyAttached Another cluster holds the backend. Give this cluster a backend of its own, or delete the holder. See A second cluster on a held backend.
False WaitingForHandover Something else still writes the backend. Wait, or remove what the message names. See Waiting for a handover.
False InvalidReference / MissingSecret The suspension ended while a reference check fails. The message ends with stay at zero until the reference check passes. Correct the reference that the message names.
False VersionDowngradeRefused The cluster resumed on a version below the one that its brokers run. Set the version forward, or downgrade on purpose.
False VersionSkipRefused The cluster resumed on a version more than one minor above the version that its brokers run. Set the version to the next minor first. See A skipped minor is refused.

Suspension holds

A suspension hold is an annotation whose key starts with suspension-hold.camunda.io/. A cluster that carries at least one hold stays suspended, whatever spec.suspend says. The value of the annotation says who holds the cluster, and why.

A restore puts a hold on its target, so that the target cannot start while the restore writes its storage. The restore removes the hold when it completes, or when you delete it and its work has stopped. A failed restore keeps its hold. The restore pages say when each restore lets go.

apiVersion: core.camunda.io/v1
kind: CamundaCluster
metadata:
  name: my-cluster
  namespace: my-cluster-ns
  annotations:
    suspension-hold.camunda.io/0d9c2a41-5e0b-4c7f-9a53-2f1e8b6c7d10: LogicalRestoreRDBMS my-cluster-ns/my-restore restores into this cluster
# ... the rest of your cluster

A held cluster reports SuspensionHeld, and the message names each hold. If a reference check fails, Ready reports that failure instead.

status:
  conditions:
    - type: Ready
      status: "False"
      reason: SuspensionHeld
      message: >-
        The cluster stays suspended while it carries a suspension hold,
        whatever spec.suspend says. Holds:
        suspension-hold.camunda.io/0d9c2a41-5e0b-4c7f-9a53-2f1e8b6c7d10
        (LogicalRestoreRDBMS my-cluster-ns/my-restore restores into this cluster)

A change to spec.suspend does not remove a hold. To end the hold of a failed restore, delete the restore, or remove the annotation by hand. If the restore also set spec.suspend, the cluster stays suspended until you clear it.

CAUTION: Do not remove the hold of a restore that still runs. If spec.suspend is also false, the brokers can start over storage that the restore writes. The restore then reports ClusterNotSuspended and fails ten minutes later. Suspend the cluster again before then.

Deletion

Deleting the cluster removes every resource that the operator created for it, and it releases the backend that the cluster held. Another cluster on that backend starts when the pods of the deleted one are gone. The broker volumes follow spec.zeebe.persistentVolumeClaimRetentionPolicy.whenDeleted (see Storage). The ElasticsearchCluster, the Database, and the contracts are separate resources and stay. A ServiceAccount with create: false stays too, also when the operator created it before the switch.

Status

kubectl get camundacluster shows Ready, its reason, the Camunda version, and the age. A suspended cluster shows no version.

Ready is True only when every process that the cluster needs is True. Its reason and message come from the first process that is not True. An embedded gateway or web application, and disabled connectors, read True with reason Disabled and stay out of Ready.

Type Reason Meaning What to do
ZeebeReady Healthy Every broker replica is ready. Nothing.
GatewayReady Healthy / Disabled Every gateway replica is ready, or the gateway is embedded. Nothing.
OperateReady / TasklistReady / AdminReady Healthy / Disabled The standalone web application is ready, or it is embedded. Nothing.
ConnectorsReady Healthy / Disabled Every connectors replica is ready, or connectors are not enabled. Nothing.
AdminSecretReady Healthy / Disabled The Secret <name>-camunda-admin is applied, or the cluster uses OIDC. Nothing.
AdminSecretReady ConnectionFailed A change of the password or the email of the admin user is not applied yet, because the cluster did not answer. The Secret keeps the active password. The operator tries again. It clears when the gateway answers.
AdminSecretReady InvalidCredentials A change of the admin user is not applied yet, because the cluster refused the password that the Secret publishes. The operator tries again. Set the password from the Secret on the admin user in the Admin web application.
AdminSecretReady Rejected A change of the admin user is not applied yet, because the cluster refused the call itself. Read the message, which names the reason.
MirroredSecretsReady Healthy / Disabled Every copy of a Secret that the CamundaPlatformConfig names is applied, or no such Secret exists. Nothing.
Ready Healthy Every process that the cluster needs is healthy. Nothing.
Ready Creating / Updating / Scaling A process rolls out or scales. The reason stays while a replica does not become ready, until the grace period ends, 30 minutes by default. Wait. If the reason stays, read the pods and events of the process that the message names.
Ready Failing A process has replicas that do not become ready. Read the pods of the named process.
Ready Degraded A process is still not ready at the end of its grace period, but some of its replicas are ready. Read the pods and events of the process that the message names.
Ready Down A process is still not ready at the end of its grace period, and none of its replicas is ready. Read the pods and events of the process that the message names.
Ready Suspended / SuspensionHeld / StorageAlreadyAttached / WaitingForHandover Every workload is at zero. Only Suspended has Ready: True. See Why the cluster is at zero replicas.
Ready InvalidReference A referenced resource does not exist, or the merged spec is invalid. Other causes are an absent ServiceAccount with create: false, two conflicting buckets, a shared Azure container, or a missing snapshot repository. A connectors ServiceAccount with the name of the cluster ServiceAccount is a cause too. So are connectors without their own ServiceAccount when the cluster ServiceAccount is default. A running cluster keeps its workloads. Read the message. Create the missing resource or correct the named field.
Ready MissingSecret A referenced Secret or one of its keys is missing. A running cluster keeps its workloads. Create the Secret with the named key.
Ready VersionDowngradeRefused The effective version is below the version that the brokers run, and no annotation sanctions the move. The operator applies nothing. Set the version forward again, or downgrade on purpose.
Ready VersionSkipRefused The effective version is more than one minor above the version that the brokers run. The operator applies nothing. Move one minor at a time. Set the version to the latest patch of the next minor first, see A skipped minor is refused.

When the operator cannot apply a workload of a cluster in StorageAlreadyAttached or WaitingForHandover, the message also carries the last apply error.

status.gateway publishes the address of the client APIs. It is empty while the cluster is suspended.

status:
  gateway:
    # The Service of the process that runs the gateway, on port 26500. No scheme: a Zeebe client takes the address in this form.
    grpcEndpoint: my-cluster-gateway.my-cluster-ns.svc:26500
    # The same Service, on port 8080. The base URL of the /v2 REST API.
    restEndpoint: http://my-cluster-gateway.my-cluster-ns.svc:8080

status.management publishes the address of the management API, which the backup kinds call. It is empty while the cluster is suspended.

status:
  management:
    # The Service of the process that serves the management API, on port 9600.
    endpoint: http://my-cluster-zeebe.my-cluster-ns.svc:9600
    auth:
      # none | basic. Camunda 8.9 serves the management port without authentication, so the operator reports none.
      method: none
    version: 8.9.9
    partitions: 3
    # Elasticsearch path with a backupStorageRef only: the snapshot repository of the storage contract.
    backupRepository: my-cluster-ns.my-cluster-es

status.adminPassword.rotation is the last passwordRotation value that the operator applied, after the preset merge. A rotation is in progress while the effective value is not empty and differs from it. A cluster that inherits the value from its preset has none in its own spec, so compare the preset value with the status.

status.serviceAccountName is the ServiceAccount that the pods run under, except the connectors, or empty for the default account of the namespace. A backup Job runs under the same account.

status.connectorsServiceAccountName is the ServiceAccount that the connectors run under. It is empty when neither the cluster nor its preset sets connectors.serviceAccount, and when connectors are not enabled.

status:
  # Empty when the pods run under the default ServiceAccount of the namespace.
  serviceAccountName: my-cluster-camunda
  # Set only when the cluster or its preset sets connectors.serviceAccount, while connectors are enabled.
  connectorsServiceAccountName: my-cluster-connectors

status.volumes lists every bound broker volume, sorted by name, with its name and its capacity. status.observedGeneration is the last generation the operator reconciled.

Routes

status.endpoints lists the routes of the cluster, sorted by name. Each entry is a path on the external URL, with the Service and the port that serve it. Read this list to route traffic to the cluster. You do not have to work out from the mode fields which process serves a path. The cluster from Topology, with Operate standalone and connectors enabled, publishes:

status:
  endpoints:
  - name: admin
    pathPrefix: /admin
    port: 8080
    protocol: HTTP
    # Admin is embedded, so the process that runs the gateway serves it.
    service: my-cluster-gateway
  - name: connectors
    # Only while spec.connectors.enabled is true.
    pathPrefix: /inbound
    port: 8080
    protocol: HTTP
    service: my-cluster-connectors
  - name: default
    # Every path that no other entry names, for example the login page.
    pathPrefix: /
    port: 8080
    protocol: HTTP
    service: my-cluster-gateway
  - name: grpc
    # No path prefix. gRPC needs a route of its own.
    port: 26500
    protocol: GRPC
    service: my-cluster-gateway
  - name: operate
    pathPrefix: /operate
    port: 8080
    protocol: HTTP
    # Operate is standalone, so it has its own Service.
    service: my-cluster-operate
  - name: rest
    pathPrefix: /v2
    port: 8080
    protocol: HTTP
    service: my-cluster-gateway
  - name: tasklist
    pathPrefix: /tasklist
    port: 8080
    protocol: HTTP
    service: my-cluster-gateway

Match the longest pathPrefix first. Thus /operate goes to its own Service, and / takes the paths that are left. An embedded web application names the Service of the process that runs the gateway, as admin and tasklist do above. With an embedded gateway, every entry that names my-cluster-gateway above names my-cluster-zeebe-client instead.

Camunda serves the inbound webhooks of connectors under /inbound/<webhook ID>. See HTTP Webhook connector.

The entries stay while the cluster is suspended, and while a reference check fails. The Services stay too, so a route answers again when the cluster resumes. status.gateway and status.management are empty while the cluster is suspended.

Fields

The API reference lists every field of CamundaCluster, with its type, whether it is required, its default, and its validation.

Validation rules

The API server enforces these rules at admission:

  • spec.storageRef and spec.platformConfigRef are required.
  • spec.version and spec.connectors.version must be of the form x.y.z.
  • spec.zeebe.partitions cannot be decreased, and once set it cannot be removed.
  • spec.zeebe.storageClassName is immutable.
  • A new inline spec.zeebe.storageSize cannot be smaller than the inline value it replaces.
  • spec.zeebe.replicationFactor must not exceed spec.zeebe.replicas.
  • spec.connectors.serviceAccount.name must differ from spec.serviceAccount.name.
  • spec.connectors.serviceAccount.name: default requires create: false.
  • spec.zeebe.persistentVolumeClaimRetentionPolicy.whenDeleted is Delete or Retain.
  • An extraEnv entry sets value or valueFrom, never both.
  • spec.auth.basic.adminEmail is empty or an address with a dot in its domain.
  • spec.auth.clientId requires spec.auth.clientSecretRef.
  • spec.backup.dump.scratchVolume.storageClassName requires sizeLimit.
  • spec.backup.primaryStorage.checkpointInterval and retention.window are ISO 8601 durations of days and time. Weeks, months, and years are rejected.

The operator checks these rules on the merged spec after the preset and the release are applied. When one fails, it reports Ready: InvalidReference with a message that starts with invalid effective spec:.

  • The effective version is present and 8.9.0 or later.
  • The effective replicationFactor does not exceed the effective replicas, and the effective partitions is at least 1.
  • connectors.version is present when connectors are enabled.
  • backup.primaryStorage.continuous is not true with a schedule of none.
  • An extraEnv entry under CAMUNDA_SECURITY_INITIALIZATION_DEFAULTROLES_ reads as <role>_<type>_<n> or <role>_<type>, with USERS, CLIENTS, GROUPS, ROLES, or MAPPINGRULES as the type. MAPPING_RULES and MAPPING-RULES also work. Camunda stops the identity initialization on any other form. The brokers alone run the identity initialization. So the rule covers the top level, zeebe, and each block that runs embedded on the brokers. Authentication shows the form that works.

Two separate rules compare the effective version with the version that the brokers run. A lower version is refused with reason VersionDowngradeRefused, see A lower version is refused. A version that skips a minor is refused with reason VersionSkipRefused, see A skipped minor is refused.

A production-shaped example

apiVersion: core.camunda.io/v1
kind: CamundaCluster
metadata:
  name: my-cluster
  namespace: my-cluster-ns
spec:
  platformConfigRef: "my-platform-config"
  presetRef: "medium"
  releaseRef: "camunda-8-9-4"
  externalUrl: "https://my-cluster.camunda.example.com"
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: "arn:aws:iam::123456789012:role/my-cluster-role"
  zeebe:
    replicas: 5
    resources:
      requests:
        memory: "8Gi"
      limits:
        memory: "8Gi"
    extraEnv:
      - name: JAVA_TOOL_OPTIONS
        value: "-XX:+ExitOnOutOfMemoryError -Xmx2g"
  storageRef: "my-storage-config"
  backupStorageRef: "my-backup-bucket"
  monitoring:
    serviceMonitor:
      enabled: true
      labels:
        prometheus: "platform"