Secondary storage

Camunda keeps its data in two places. Primary storage is the Zeebe log and state on the broker volumes. Secondary storage holds the exported data that Operate, Tasklist, and the search API read, and Camunda describes it in Configure secondary storage. The operator supports two backends: Elasticsearch through the ECK operator, and PostgreSQL through the CloudNativePG operator or on a server you run. A CamundaCluster does not name the backend. It names a SecondaryStorageConfig in spec.storageRef, and this contract points at the backend.

This guide tells you which backend to pick, which resources to create for each backend, and in which order.

Choose a backend

	Elasticsearch	PostgreSQL
Who runs the backend	The ECK operator, in your Kubernetes cluster. You install ECK first.	The CloudNativePG operator, or you. See PostgreSQL.
What you create	One ElasticsearchCluster.	One DatabaseServer and one Database, or a DatabaseServerConfig for your own server and one Database.
Versions	Elasticsearch 8.19+ and 9.2+.	See The PostgreSQL version.
Backup	Elasticsearch snapshots, through LogicalBackupElasticsearch.	A pg_dump of the database plus a Zeebe backup of primary storage, through LogicalBackupRDBMS.
Optimize	Supported.	Not supported. Camunda Optimize needs Elasticsearch or OpenSearch.
Not supported by the operator	OpenSearch. The operator does not model it.	Other database engines. DatabaseServerConfig accepts engine: postgres only.

If you need Optimize, pick Elasticsearch. If you already run PostgreSQL and do not need Optimize, you can pick PostgreSQL. Camunda compares the throughput and the sizing of the backends in Size your environment.

One cluster per backend

A backend belongs to one CamundaCluster. Camunda fixes the index names in Elasticsearch and the tables in a database, so two clusters on one backend write each other's data. Give every cluster its own ElasticsearchCluster or its own Database, each with its own contract. Two clusters can share one PostgreSQL server, each with its own database, see One server per orchestration cluster.

If a second cluster names a backend that another cluster holds, the operator keeps the second cluster suspended. Its Ready condition reads False with reason StorageAlreadyAttached. The CamundaCluster reference has the rule in full.

Elasticsearch

Prerequisite: the ECK operator is installed in the Kubernetes cluster, and the camunda-operator started after the ECK CRDs were installed. The operator does not run Elasticsearch nodes itself. It creates an ECK Elasticsearch resource, and ECK runs the nodes.

This chain is ready to apply in config/example/camunda-cluster/elasticsearch. The link opens the main branch. For a release, open the same path under the tag of the release.

1. Create an ElasticsearchCluster. The sizes below are for trying out. For production, use more nodes and larger volumes. On one node, the Camunda indices get no replica, so the node keeps green health. Node count has the rule.

   apiVersion: core.camunda.io/v1
   kind: ElasticsearchCluster
   metadata:
     name: my-cluster-es
     namespace: my-cluster-ns
   spec:
     version: "9.2.4"
     replicas: 1
     storageSize: "16Gi"
     resources:
       requests: { cpu: "500m", memory: "1Gi" }
     secondaryStorageConfig: "my-storage-config"
   

2. Wait until the ElasticsearchCluster is Ready with reason Healthy.

   kubectl wait elasticsearchcluster/my-cluster-es -n my-cluster-ns --for=condition=Ready --timeout=15m
   

   The first start takes some minutes while ECK pulls the image and forms the cluster. When it is done, the status reads:

   status:
     conditions:
       - type: Ready
         status: "True"
         reason: Healthy
   

   If Ready stays False, read the message of the Ready condition with kubectl get elasticsearchcluster my-cluster-es -n my-cluster-ns -o yaml.

3. Make sure that the SecondaryStorageConfig my-storage-config exists in the same namespace and is Ready with reason Healthy. The operator created it for you.

   kubectl get secondarystorageconfig my-storage-config -n my-cluster-ns
   

   NAME                READY   REASON    TYPE            AGE
   my-storage-config   True    Healthy   elasticsearch   2m
   

4. Point the CamundaCluster at it with spec.storageRef: my-storage-config. The CamundaCluster must be in the same namespace as the SecondaryStorageConfig, because storageRef is a name without a namespace.

   apiVersion: core.camunda.io/v1
   kind: CamundaCluster
   metadata:
     name: my-cluster
     namespace: my-cluster-ns
   spec:
     version: "8.9.9"
     platformConfigRef: "my-platform-config"
     storageRef: "my-storage-config"
   

graph LR
    ESC[ElasticsearchCluster] -->|creates| ECK["Elasticsearch (ECK operator, external)"]
    ESC -->|creates| SEC["Secret my-cluster-es-es-user"]
    ESC -->|creates| SSC[SecondaryStorageConfig]
    CC[CamundaCluster] -.->|storageRef| SSC

Hold "Alt" / "Option" to enable pan & zoom

The contract carries the endpoint, the user Secret my-cluster-es-es-user, and the CA of the self-signed certificate. For backups, also set spec.snapshotStorageRef, see Backup. ElasticsearchCluster has the credentials and the status, and links every field.

PostgreSQL

The chain has two halves. The first half is the server, and you get it in one of two ways. The second half is the logical database on that server, and it is the same either way.

config/example/camunda-cluster/rdbms on the main branch is ready to apply. It takes the DatabaseServer route, and it holds the Database, the platform configuration, and the CamundaCluster.

	A DatabaseServer	A server you run
Who runs PostgreSQL	The CloudNativePG operator, in your Kubernetes cluster.	You. The server can run in the cluster or outside it, for example as a managed service.
What you create	One DatabaseServer.	One Secret with the admin credentials and one DatabaseServerConfig.
Admin credentials	CloudNativePG writes them into a Secret.	You write them.
Point-in-time restore	spec.archive gives the server a continuous archive, and a PointInTimeRestore rolls the server back for you.	Only if you archive the server yourself and roll it back before you create the restore.
Prerequisite	The CloudNativePG operator. An archive also needs the Barman Cloud plugin and cert-manager.	Nothing on the Kubernetes cluster.

One server per orchestration cluster

Give every orchestration cluster a PostgreSQL server of its own. Two clusters can share one server with a database each, and the operator allows it. A shared server gives up two things.

A point-in-time restore rolls back the whole server, not one database. Every database on it goes back to the same point. A restore for one cluster therefore erases the recent data of every other cluster on that server. A PointInTimeRestore holds in Pending while more than one Database uses the server. Its Ready condition reads False with reason SharedServer. The hold lifts when one Database is left.

Sizing and the version are shared too. One DatabaseServer has one instance count, one set of volumes, and one PostgreSQL major. A change to any of them reaches every cluster on the server. When a change restarts the instances, every cluster on the server sees that restart. The major cannot change once the server runs, see The PostgreSQL version.

A shared server is cheaper, and it fits a fleet of small clusters that never needs a point-in-time restore. Choose it only when you accept both losses above.

Run the server with a DatabaseServer

Prerequisites: the CloudNativePG operator is installed in the Kubernetes cluster, and the camunda-operator started after its CRDs were installed. An archive also needs the Barman Cloud plugin and cert-manager. See Installation.

1. Create a DatabaseServer. The sizes below are for production. Leave out archive when you do not need a point-in-time restore.

   apiVersion: core.camunda.io/v1
   kind: DatabaseServer
   metadata:
     name: my-db
     namespace: my-cluster-ns
   spec:
     version: "17"
     instances: 2
     storageSize: "64Gi"
     resources:
       requests: { cpu: "1", memory: "2Gi" }
     databaseServerConfig: my-db-server
     archive:
       objectStorageRef: my-backup-bucket
       retentionPeriodDays: 30
   

   objectStorageRef names an ObjectStorageConfig. The backup guide creates one.

2. Wait until the DatabaseServer is Ready with reason Healthy.

   kubectl wait databaseserver/my-db -n my-cluster-ns --for=condition=Ready --timeout=15m
   

   The first start pulls the PostgreSQL image, bootstraps the instances, and takes the first base backup of the archive. When it is done, the status reads:

   status:
     cluster: my-db
     systemIdentifier: "7412345678901234567"
     conditions:
       - type: Ready
         status: "True"
         reason: Healthy
   

   The server publishes the DatabaseServerConfig my-db-server in the same namespace, with the host my-db-rw.my-cluster-ns.svc, port 5432, and the Secret my-db-superuser as its admin credentials. With an archive it also carries pitr.enabled: true and pitr.recovery: operator.

   DatabaseServer covers sizing, the archive, monitoring, presets, and the rollback in full.

3. Continue with The logical database. Use serverRef: my-db-server.

Use a server that you run

Prerequisites: a PostgreSQL server that the operator can reach over the network, and an admin user that can create databases and roles. The steps below use a server at postgres.my-cluster-ns.svc on port 5432 and an admin user postgres.

1. Create a Secret with the admin credentials in the namespace of the CamundaCluster. The DatabaseServerConfig names it by name, and reads it from its own namespace.

   apiVersion: v1
   kind: Secret
   metadata:
     name: my-db-server-admin-credentials
     namespace: my-cluster-ns
   type: Opaque
   stringData:
     username: postgres
     password: change-me
   

2. Create a DatabaseServerConfig in the namespace of the cluster. It describes the server and names the admin credentials Secret beside it.

   apiVersion: core.camunda.io/v1
   kind: DatabaseServerConfig
   metadata:
     name: my-db-server
     namespace: my-cluster-ns
   spec:
     engine: postgres
     host: "postgres.my-cluster-ns.svc"
     port: 5432
     adminCredentialsSecretRef:
       name: my-db-server-admin-credentials
   

3. Wait until the DatabaseServerConfig is Ready with reason Healthy. The operator connects to the server with the admin credentials. It reports the major version and the identity of the instance:

   status:
     serverVersion: "17"
     systemIdentifier: "7412345678901234567"
     conditions:
       - type: Ready
         status: "True"
         reason: Healthy
         message: Reached the server; it runs major version 17
   

   If the operator cannot reach the server, or the server rejects the credentials, the reason is ConnectionFailed. The message names the endpoint and the error:

   status:
     conditions:
       - type: Ready
         status: "False"
         reason: ConnectionFailed
         message: "Connecting to postgres.my-cluster-ns.svc:5432: failed to connect: password authentication failed for user \"postgres\""
   

   If the reason is MissingSecret, the Secret or one of its keys does not exist.

The logical database

Both paths above end with a DatabaseServerConfig in the namespace of the cluster. The steps below turn it into secondary storage. They use the contract my-db-server.

1. Create a Database in the namespace of the CamundaCluster. Set secondaryStorageConfig to the name of the contract you want. The Database resolves serverRef in its own namespace. It publishes the DatabaseConfig and the SecondaryStorageConfig there, where the cluster reads them.

   apiVersion: core.camunda.io/v1
   kind: Database
   metadata:
     name: my-camunda-db
     namespace: my-cluster-ns
   spec:
     serverRef: "my-db-server"
     databaseName: "camunda"
     secondaryStorageConfig: "my-storage-config"
   

2. Wait until the Database is Ready with reason Healthy.

   kubectl wait database/my-camunda-db -n my-cluster-ns --for=condition=Ready --timeout=5m
   

   If the reason is ServerIdentityUnknown, the DatabaseServerConfig has not reached the server yet. Read its Ready condition.

3. Make sure that the DatabaseConfig my-camunda-db and the SecondaryStorageConfig my-storage-config exist in my-cluster-ns and are Ready with reason Healthy. The Database created both. The SecondaryStorageConfig has type: rdbms and references the DatabaseConfig.

   kubectl get databaseconfig,secondarystorageconfig -n my-cluster-ns
   

4. Point the CamundaCluster at the contract, as in step 4 of the Elasticsearch procedure:

   apiVersion: core.camunda.io/v1
   kind: CamundaCluster
   metadata:
     name: my-cluster
     namespace: my-cluster-ns
   spec:
     # ... version, platformConfigRef, and the rest of your cluster
     storageRef: my-storage-config
   

graph LR
    DBS[DatabaseServer] -->|publishes| DBSC
    DB[Database] -.->|serverRef| DBSC[DatabaseServerConfig]
    DBSC -.->|adminCredentialsSecretRef| ADM["Secret with the admin credentials"]
    DB -->|"SQL: CREATE DATABASE, CREATE ROLE"| PG["PostgreSQL server"]
    DB -->|creates| DBC[DatabaseConfig]
    DB -->|creates| SSC["SecondaryStorageConfig (rdbms)"]
    SSC -.->|databaseConfigRef| DBC
    CC[CamundaCluster] -.->|storageRef| SSC

Hold "Alt" / "Option" to enable pan & zoom

The Database also creates the database, its roles, and their credential Secrets. Database names them, and says how to rotate a password and what deletion keeps.

For each kind, see DatabaseServerConfig, Database, DatabaseConfig, and SecondaryStorageConfig.

Bring your own backend

If you already run Elasticsearch or a PostgreSQL database that the operator does not manage, write the contracts by hand. The CamundaCluster does not know who created them.

The rule of one cluster per backend holds for a hand-written contract. Write one contract per backend.

For Elasticsearch, write a SecondaryStorageConfig with type: elasticsearch. Create the Secret with the username and password first. The endpoint can serve a certificate that the orchestration cluster does not trust by default. In that case, name the CA Secret as well.

apiVersion: core.camunda.io/v1
kind: SecondaryStorageConfig
metadata:
  name: my-storage-config
  namespace: my-cluster-ns
spec:
  type: elasticsearch
  elasticsearch:
    endpoint: "https://elasticsearch.example.com:9200"
    credentialsSecretRef:
      name: my-elasticsearch-credentials
      usernameKey: username
      passwordKey: password
    # Only when the certificate of the endpoint is not trusted by default.
    caSecretRef:
      name: my-elasticsearch-ca
      key: ca.crt
    # The number of data nodes. It sets the default index replica count.
    nodeCount: 3

Without nodeCount, Camunda keeps its own default replica count, see Node count.

For PostgreSQL, write a DatabaseServerConfig for the server, a DatabaseConfig for the database, and a SecondaryStorageConfig with type: rdbms. All three live in the namespace of the cluster. Create the database, the application role, and the credentials Secret on your side first. The DatabaseServerConfig still needs admin credentials, because the operator validates it by a connection to the server.

apiVersion: core.camunda.io/v1
kind: DatabaseConfig
metadata:
  name: my-camunda-db
  namespace: my-cluster-ns
spec:
  serverRef: my-db-server
  databaseName: camunda
  credentialsSecretRef:
    name: my-camunda-db-credentials
    usernameKey: username
    passwordKey: password
---
apiVersion: core.camunda.io/v1
kind: SecondaryStorageConfig
metadata:
  name: my-storage-config
  namespace: my-cluster-ns
spec:
  type: rdbms
  rdbms:
    databaseConfigRef: my-camunda-db

The reference pages describe each contract: SecondaryStorageConfig, DatabaseConfig, and DatabaseServerConfig. A backup of a hand-written Elasticsearch contract needs elasticsearch.snapshotRepository set by hand, after you register the repository yourself. A backup of a hand-written DatabaseConfig needs backupCredentialsSecretRef. The backup guide shows both.

Related

• Getting started: the full path from an empty namespace to a running orchestration cluster.
• Backup: how backups work on each backend, and which bucket both sides must share.
• SecondaryStorageConfig: the contract that a CamundaCluster references through storageRef.
• ElasticsearchCluster: the kind that runs Elasticsearch through ECK and creates the contract.
• DatabaseServer: the kind that runs PostgreSQL through CloudNativePG, archives it, and creates the contract.
• DatabaseServerConfig: the contract that describes a PostgreSQL server and its admin credentials.
• Database: the kind that prepares a logical database and creates the contracts.
• DatabaseConfig: the contract that describes one logical database and its credentials.
• CamundaCluster: the orchestration cluster, and the meaning of storageRef.