Camunda 8 self managed Multitenancy

savi003 · October 12, 2023, 3:57am

I saw articles related to multi-tenancy in camunda 8, I took the latest pull from the main branch yesterday(10th October 2023) of camunda platform (GitHub - camunda/camunda-platform: Links to Camunda Platform 8 resources, releases, and local development config), but when i started the applications and checked the identity application , there is no options for tenants as showed in the article (https://camunda.com/blog/2023/10/camunda-8-3-scaling-automation-maximize-value/#multitenancy"), Is the tenants feature not yet released?

nathan.loding · October 12, 2023, 3:33pm

Hi @savi003 - did you enable multi-tenancy?

savi003 · October 13, 2023, 12:46am

Hi @nathan.loding , No I didn’t know, I had to enable it, I am running camunda through docker, where is that option in case of docker?

nathan.loding · October 13, 2023, 3:38pm

There are environment variables for each of the services that need to be enabled. Full documentation in our docs is coming very soon, but you can find the environment variables by searching for where that value is used in the Helm charts: https://github.com/search?q=repo%3Acamunda%2Fcamunda-platform-helm%20.Values.global.multitenancy.enabled&type=code

If you set those environment variables, it should enable multi-tenancy for you!

savi003 · October 13, 2023, 4:23pm

@nathan.loding I’m not using helm in for local development, I’m using this repo GitHub - camunda/camunda-platform: Links to Camunda Platform 8 resources, releases, and local development config to run the services on docker in my local machine, is multitenancy available here?

nathan.loding · October 13, 2023, 4:34pm

@savi003 - those exact same Docker images are used for the Kubernetes cluster created by the Helm charts. The Helm chart values map to environment variables, so by searching for where the Helm charts are using a specific settings you can find the related environment variables. If you look at the example docker-compose.yml, it is setting those same environment variables.

savi003 · October 13, 2023, 4:48pm

@nathan.loding In this repo of helm I searched for multitenancy https://github.com/search?q=repo%3Acamunda%2Fcamunda-platform-helm%20multitenancy&type=code and i am seeing the configurations related to multitenancy

but when i searched for multitenancy in this repo which I’m using to run the services in my local machine GitHub - camunda/camunda-platform: Links to Camunda Platform 8 resources, releases, and local development config I’m not seeing any configurations related to multitenancy
https://github.com/search?q=repo%3Acamunda%2Fcamunda-platform%20multitenancy&type=code

nathan.loding · October 13, 2023, 5:45pm

@savi003 - that is correct. The environment variables are used by the applications, not by Docker, and not all environment variables are used in that Docker configuration.

savi003 · October 13, 2023, 5:52pm

@nathan.loding so i f want to enable multitenancy in my local with GitHub - camunda/camunda-platform: Links to Camunda Platform 8 resources, releases, and local development config what configurations do i have to make in the docker-compose.yaml (https://github.com/camunda/camunda-platform/blob/97cc173c306195eb3359e5b147b647b66f1dc60e/docker-compose.yaml) file?

nathan.loding · October 13, 2023, 5:54pm

@savi003 - you need to add all the environment variables referenced in that Helm search. Then, when you rebuild the container, the applications will see those variables and the feature will be enabled.

savi003 · October 13, 2023, 5:57pm

@nathan.loding Could you please provide me with a sample?

nathan.loding · October 13, 2023, 6:14pm

I do not have a sample Docker configuration for multi-tenancy at this time; you just need to add those environment variables for each application, the same way the environment variables are currently used in the docker-compose.yaml file.

For instance, under services.zeebe.environment in the docker-compose.yaml file, you need to add:

ZEEBE_GATEWAY_MULTITENANCY_ENABLED=true
ZEEBE_BROKER_GATEWAY_MULTITENANCY_ENABLED=true

You can find all the environment variables from the Helm charts in the link I provided.

savi003 · October 14, 2023, 2:09am

@nathan.loding I have added the environment variables in helm to the docker-compose file, now I’m getting an error in zeebe sayings authorized tenants is null

2023-10-14 07:33:26 2023-10-14 02:03:26.233 [] [grpc-executor-0] [] ERROR
2023-10-14 07:33:26       io.camunda.zeebe.gateway - Expected to handle gRPC request, but an unexpected error occurred
2023-10-14 07:33:26 java.lang.NullPointerException: Cannot invoke "java.util.List.contains(Object)" because "authorizedTenants" is null
2023-10-14 07:33:26     at io.camunda.zeebe.gateway.RequestMapper.ensureTenantIdSet(RequestMapper.java:334) ~[zeebe-gateway-8.3.0.jar:8.3.0]
2023-10-14 07:33:26     at io.camunda.zeebe.gateway.RequestMapper.lambda$ensureTenantIdsSet$0(RequestMapper.java:354) ~[zeebe-gateway-8.3.0.jar:8.3.0]
2023-10-14 07:33:26     at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(Unknown Source) ~[?:?]
2023-10-14 07:33:26     at java.util.stream.ReferencePipeline$Head.forEach(Unknown Source) ~[?:?]
2023-10-14 07:33:26     at io.camunda.zeebe.gateway.RequestMapper.ensureTenantIdsSet(RequestMapper.java:354) ~[zeebe-gateway-8.3.0.jar:8.3.0]
2023-10-14 07:33:26     at io.camunda.zeebe.gateway.RequestMapper.toActivateJobsRequest(RequestMapper.java:220) ~[zeebe-gateway-8.3.0.jar:8.3.0]
2023-10-14 07:33:26     at io.camunda.zeebe.gateway.EndpointManager.mapToBrokerRequest(EndpointManager.java:464) ~[zeebe-gateway-8.3.0.jar:8.3.0]
2023-10-14 07:33:26     at io.camunda.zeebe.gateway.EndpointManager.activateJobs(EndpointManager.java:183) [zeebe-gateway-8.3.0.jar:8.3.0]
2023-10-14 07:33:26     at io.camunda.zeebe.gateway.GatewayGrpcService.activateJobs(GatewayGrpcService.java:63) [zeebe-gateway-8.3.0.jar:8.3.0]
2023-10-14 07:33:26     at io.camunda.zeebe.gateway.protocol.GatewayGrpc$MethodHandlers.invoke(GatewayGrpc.java:2022) [zeebe-gateway-protocol-impl-8.3.0.jar:8.3.0]
2023-10-14 07:33:26     at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182) [grpc-stub-1.58.0.jar:1.58.0]
2023-10-14 07:33:26     at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35) [grpc-api-1.58.0.jar:1.58.0]
2023-10-14 07:33:26     at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23) [grpc-api-1.58.0.jar:1.58.0]
2023-10-14 07:33:26     at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:351) [grpc-core-1.58.0.jar:1.58.0]
2023-10-14 07:33:26     at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:860) [grpc-core-1.58.0.jar:1.58.0]
2023-10-14 07:33:26     at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.58.0.jar:1.58.0]
2023-10-14 07:33:26     at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) [grpc-core-1.58.0.jar:1.58.0]
2023-10-14 07:33:26     at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(Unknown Source) [?:?]
2023-10-14 07:33:26     at java.util.concurrent.ForkJoinTask.doExec(Unknown Source) [?:?]
2023-10-14 07:33:26     at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source) [?:?]
2023-10-14 07:33:26     at java.util.concurrent.ForkJoinPool.scan(Unknown Source) [?:?]
2023-10-14 07:33:26     at java.util.concurrent.ForkJoinPool.runWorker(Unknown Source) [?:?]
2023-10-14 07:33:26     at java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source) [?:?]

this is my modified docker-compose file

# While the Docker images themselves are supported for production usage,
# this docker-compose.yaml is designed to be used by developers to run
# an environment locally. It is not designed to be used in production.
# We recommend to use Kubernetes in production with our Helm Charts:
# https://docs.camunda.io/docs/self-managed/platform-deployment/kubernetes-helm/
# For local development, we recommend using KIND instead of `docker-compose`:
# https://docs.camunda.io/docs/self-managed/platform-deployment/helm-kubernetes/guides/local-kubernetes-cluster/

# This is a full configuration with Zeebe, Operate, Tasklist, Optimize, Identity, Keycloak, and Elasticsearch
# See docker-compose-core.yml for a lightweight configuration that does not include Optimize, Identity, and Keycloak.

services:

  zeebe: # https://docs.camunda.io/docs/self-managed/platform-deployment/docker/#zeebe
    image: camunda/zeebe:${CAMUNDA_PLATFORM_VERSION}
    container_name: zeebe
    ports:
      - "26500:26500"
      - "9600:9600"
    environment: # https://docs.camunda.io/docs/self-managed/zeebe-deployment/configuration/environment-variables/
      - ZEEBE_BROKER_GATEWAY_SECURITY_AUTHENTICATION_MODE=${ZEEBE_AUTHENTICATION_MODE}
      - ZEEBE_BROKER_GATEWAY_SECURITY_AUTHENTICATION_IDENTITY_ISSUERBACKENDURL=http://keycloak:8080/auth/realms/camunda-platform
      - ZEEBE_BROKER_GATEWAY_SECURITY_AUTHENTICATION_IDENTITY_AUDIENCE=zeebe-api
      - ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_CLASSNAME=io.camunda.zeebe.exporter.ElasticsearchExporter
      - ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_URL=http://elasticsearch:9200
      # default is 1000, see here: https://github.com/camunda/zeebe/blob/16b482d9205a15c0ec07ae4d86dc87f6ba9e79db/exporters/elasticsearch-exporter/src/main/java/io/camunda/zeebe/exporter/ElasticsearchExporterConfiguration.java#L259
      - ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_BULK_SIZE=1
      # allow running with low disk space
      - ZEEBE_BROKER_DATA_DISKUSAGECOMMANDWATERMARK=0.998
      - ZEEBE_BROKER_DATA_DISKUSAGEREPLICATIONWATERMARK=0.999
      - ZEEBE_GATEWAY_MULTITENANCY_ENABLED=${MULTITENANCY_ENABLED}
      - ZEEBE_BROKER_GATEWAY_MULTITENANCY_ENABLED=${MULTITENANCY_ENABLED}
      - "JAVA_TOOL_OPTIONS=-Xms512m -Xmx512m"
    restart: always
    healthcheck:
      test: [ "CMD-SHELL", "timeout 10s bash -c ':> /dev/tcp/127.0.0.1/9600' || exit 1" ]
      interval: 30s
      timeout: 5s
      retries: 5
      start_period: 30s
    volumes:
      - zeebe:/usr/local/zeebe/data
    networks:
      - camunda-platform
    depends_on:
      - elasticsearch
      - identity

  operate: # https://docs.camunda.io/docs/self-managed/platform-deployment/docker/#operate
    image: camunda/operate:${CAMUNDA_PLATFORM_VERSION}
    container_name: operate
    ports:
      - "8081:8080"
    environment: # https://docs.camunda.io/docs/self-managed/operate-deployment/configuration/
      - CAMUNDA_OPERATE_ZEEBE_GATEWAYADDRESS=zeebe:26500
      - ZEEBE_CLIENT_ID=${ZEEBE_CLIENT_ID}
      - ZEEBE_CLIENT_SECRET=${ZEEBE_CLIENT_SECRET}
      - ZEEBE_TOKEN_AUDIENCE=zeebe-api
      - ZEEBE_AUTHORIZATION_SERVER_URL=http://keycloak:8080/auth/realms/camunda-platform/protocol/openid-connect/token
      - CAMUNDA_OPERATE_ELASTICSEARCH_URL=http://elasticsearch:9200
      - CAMUNDA_OPERATE_ZEEBEELASTICSEARCH_URL=http://elasticsearch:9200
      # For more information regarding configuration with Identity see:
      # https://docs.camunda.io/docs/self-managed/operate-deployment/authentication/#identity
      - SPRING_PROFILES_ACTIVE=identity-auth
      - CAMUNDA_OPERATE_IDENTITY_BASEURL=http://identity:8084
      - CAMUNDA_OPERATE_IDENTITY_ISSUER_URL=http://${HOST}:18080/auth/realms/camunda-platform
      - CAMUNDA_OPERATE_IDENTITY_ISSUER_BACKEND_URL=http://keycloak:8080/auth/realms/camunda-platform
      - CAMUNDA_OPERATE_IDENTITY_CLIENTID=operate
      - CAMUNDA_OPERATE_IDENTITY_CLIENTSECRET=XALaRPl5qwTEItdwCMiPS62nVpKs7dL7
      - CAMUNDA_OPERATE_IDENTITY_AUDIENCE=operate-api
      - SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_ISSUER_URI=http://${HOST}:18080/auth/realms/camunda-platform
      - SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_JWK_SET_URI=http://${HOST}:18080/auth/realms/camunda-platform/protocol/openid-connect/certs
      - CAMUNDA_OPERATE_IDENTITY_RESOURCEPERMISSIONSENABLED=${RESOURCE_AUTHORIZATIONS_ENABLED}
      - CAMUNDA_OPERATE_MULTITENANCY_ENABLED=${MULTITENANCY_ENABLED}
      - management.endpoints.web.exposure.include=health
      - management.endpoint.health.probes.enabled=true
    healthcheck:
      test: [ "CMD-SHELL", "curl -f http://localhost:8080/actuator/health/readiness" ]
      interval: 30s
      timeout: 1s
      retries: 5
      start_period: 30s
    networks:
      - camunda-platform
    depends_on:
      - zeebe
      - identity
      - elasticsearch

  tasklist: # https://docs.camunda.io/docs/self-managed/platform-deployment/docker/#tasklist
    image: camunda/tasklist:${CAMUNDA_PLATFORM_VERSION}
    container_name: tasklist
    ports:
      - "8082:8080"
    environment: # https://docs.camunda.io/docs/self-managed/tasklist-deployment/configuration/
      - CAMUNDA_TASKLIST_ZEEBE_GATEWAYADDRESS=zeebe:26500
      - ZEEBE_CLIENT_ID=${ZEEBE_CLIENT_ID}
      - ZEEBE_CLIENT_SECRET=${ZEEBE_CLIENT_SECRET}
      - ZEEBE_TOKEN_AUDIENCE=zeebe-api
      - ZEEBE_AUTHORIZATION_SERVER_URL=http://keycloak:8080/auth/realms/camunda-platform/protocol/openid-connect/token
      - CAMUNDA_TASKLIST_ELASTICSEARCH_URL=http://elasticsearch:9200
      - CAMUNDA_TASKLIST_ZEEBEELASTICSEARCH_URL=http://elasticsearch:9200
      # For more information regarding configuration with Identity see:
      # https://docs.camunda.io/docs/self-managed/tasklist-deployment/authentication/#identity
      - SPRING_PROFILES_ACTIVE=identity-auth
      - CAMUNDA_TASKLIST_IDENTITY_BASEURL=http://identity:8084
      - CAMUNDA_TASKLIST_IDENTITY_ISSUER_URL=http://${HOST}:18080/auth/realms/camunda-platform
      - CAMUNDA_TASKLIST_IDENTITY_ISSUER_BACKEND_URL=http://keycloak:8080/auth/realms/camunda-platform
      - CAMUNDA_TASKLIST_IDENTITY_CLIENTID=tasklist
      - CAMUNDA_TASKLIST_IDENTITY_CLIENTSECRET=XALaRPl5qwTEItdwCMiPS62nVpKs7dL7
      - CAMUNDA_TASKLIST_IDENTITY_AUDIENCE=tasklist-api
      - SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_ISSUER_URI=http://${HOST}:18080/auth/realms/camunda-platform
      - SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_JWK_SET_URI=http://${HOST}:18080/auth/realms/camunda-platform/protocol/openid-connect/certs
      - CAMUNDA_TASKLIST_IDENTITY_RESOURCE_PERMISSIONS_ENABLED=${RESOURCE_AUTHORIZATIONS_ENABLED}
      - CAMUNDA_TASKLIST_MULTITENANCY_ENABLED=${MULTITENANCY_ENABLED}
      - management.endpoints.web.exposure.include=health
      - management.endpoint.health.probes.enabled=true
    healthcheck:
      test: [ "CMD-SHELL", "curl -f http://localhost:8080/actuator/health/readiness" ]
      interval: 30s
      timeout: 1s
      retries: 5
      start_period: 30s
    networks:
      - camunda-platform
    depends_on:
      zeebe:
        condition: service_started
      elasticsearch:
        condition: service_healthy
      identity:
        condition: service_healthy

  connectors: # https://docs.camunda.io/docs/components/integration-framework/connectors/out-of-the-box-connectors/available-connectors-overview/
    image: camunda/connectors-bundle:${CAMUNDA_CONNECTORS_VERSION}
    container_name: connectors
    ports:
      - "8085:8080"
    environment:
      - ZEEBE_CLIENT_BROKER_GATEWAY-ADDRESS=zeebe:26500
      - ZEEBE_CLIENT_SECURITY_PLAINTEXT=true
      - ZEEBE_CLIENT_ID=${ZEEBE_CLIENT_ID}
      - ZEEBE_CLIENT_SECRET=${ZEEBE_CLIENT_SECRET}
      - ZEEBE_TOKEN_AUDIENCE=zeebe-api
      - ZEEBE_AUTHORIZATION_SERVER_URL=http://keycloak:8080/auth/realms/camunda-platform/protocol/openid-connect/token
      - CAMUNDA_OPERATE_CLIENT_KEYCLOAK-URL=http://keycloak:8080
      - CAMUNDA_OPERATE_CLIENT_CLIENT-ID=connectors
      - CAMUNDA_OPERATE_CLIENT_CLIENT-SECRET=XALaRPl5qwTEItdwCMiPS62nVpKs7dL7
      - CAMUNDA_OPERATE_CLIENT_KEYCLOAK-REALM=camunda-platform
      - CAMUNDA_OPERATE_CLIENT_URL=http://operate:8080
      - management.endpoints.web.exposure.include=health
      - management.endpoint.health.probes.enabled=true
    env_file: connector-secrets.txt
    healthcheck:
      test: [ "CMD-SHELL", "curl -f http://localhost:8080/actuator/health/readiness" ]
      interval: 30s
      timeout: 1s
      retries: 5
      start_period: 30s
    networks:
      - camunda-platform
    depends_on:
      - zeebe
      - operate
      - identity

  optimize: # https://docs.camunda.io/docs/self-managed/platform-deployment/docker/#optimize
    image: camunda/optimize:${CAMUNDA_OPTIMIZE_VERSION}
    container_name: optimize
    ports:
      - "8083:8090"
    environment: # https://docs.camunda.io/docs/self-managed/optimize-deployment/setup/installation/#available-environment-variables
      - OPTIMIZE_ELASTICSEARCH_HOST=elasticsearch
      - OPTIMIZE_ELASTICSEARCH_HTTP_PORT=9200
      - SPRING_PROFILES_ACTIVE=ccsm
      - CAMUNDA_OPTIMIZE_ZEEBE_ENABLED=true
      - CAMUNDA_OPTIMIZE_ENTERPRISE=false
      - CAMUNDA_OPTIMIZE_IDENTITY_ISSUER_URL=http://${HOST}:18080/auth/realms/camunda-platform
      - CAMUNDA_OPTIMIZE_IDENTITY_ISSUER_BACKEND_URL=http://keycloak:8080/auth/realms/camunda-platform
      - CAMUNDA_OPTIMIZE_IDENTITY_CLIENTID=optimize
      - CAMUNDA_OPTIMIZE_IDENTITY_CLIENTSECRET=XALaRPl5qwTEItdwCMiPS62nVpKs7dL7
      - CAMUNDA_OPTIMIZE_IDENTITY_AUDIENCE=optimize-api
      - CAMUNDA_OPTIMIZE_SECURITY_AUTH_COOKIE_SAME_SITE_ENABLED=false
      - CAMUNDA_OPTIMIZE_UI_LOGOUT_HIDDEN=true
      - CAMUNDA_OPTIMIZE_MULTITENANCY_ENABLED=${MULTITENANCY_ENABLED}
      - management.endpoints.web.exposure.include=health
      - management.endpoint.health.probes.enabled=true
    healthcheck:
      test: [ "CMD-SHELL", "curl -f http://localhost:8090/api/readyz" ]
      interval: 30s
      timeout: 1s
      retries: 5
      start_period: 30s
    volumes:
      - "./.optimize/environment-config.yaml:/optimize/config/environment-config.yaml"
    restart: on-failure
    networks:
      - camunda-platform
    depends_on:
      - identity
      - elasticsearch

  identity: # https://docs.camunda.io/docs/self-managed/platform-deployment/docker/#identity
    container_name: identity
    image: camunda/identity:${CAMUNDA_PLATFORM_VERSION}
    ports:
      - "8084:8084"
    environment: # https://docs.camunda.io/docs/self-managed/identity/deployment/configuration-variables/
      SERVER_PORT: 8084
      IDENTITY_RETRY_DELAY_SECONDS: 30
      KEYCLOAK_URL: http://keycloak:8080/auth
      IDENTITY_AUTH_PROVIDER_BACKEND_URL: http://keycloak:8080/auth/realms/camunda-platform
      IDENTITY_DATABASE_HOST: postgres
      IDENTITY_DATABASE_PORT: 5432
      IDENTITY_DATABASE_NAME: bitnami_keycloak
      IDENTITY_DATABASE_USERNAME: bn_keycloak
      IDENTITY_DATABASE_PASSWORD: "#3]O?4RGj)DE7Z!9SA5"
      KEYCLOAK_INIT_OPERATE_SECRET: XALaRPl5qwTEItdwCMiPS62nVpKs7dL7
      KEYCLOAK_INIT_OPERATE_ROOT_URL: http://${HOST}:8081
      KEYCLOAK_INIT_TASKLIST_SECRET: XALaRPl5qwTEItdwCMiPS62nVpKs7dL7
      KEYCLOAK_INIT_TASKLIST_ROOT_URL: http://${HOST}:8082
      KEYCLOAK_INIT_OPTIMIZE_SECRET: XALaRPl5qwTEItdwCMiPS62nVpKs7dL7
      KEYCLOAK_INIT_OPTIMIZE_ROOT_URL: http://${HOST}:8083
      KEYCLOAK_INIT_WEBMODELER_ROOT_URL: http://${HOST}:8070
      KEYCLOAK_INIT_CONNECTORS_SECRET: XALaRPl5qwTEItdwCMiPS62nVpKs7dL7
      KEYCLOAK_INIT_CONNECTORS_ROOT_URL: http://${HOST}:8085
      KEYCLOAK_INIT_ZEEBE_NAME: zeebe
      KEYCLOAK_USERS_0_USERNAME: "demo"
      KEYCLOAK_USERS_0_PASSWORD: "demo"
      KEYCLOAK_USERS_0_FIRST_NAME: "demo"
      KEYCLOAK_USERS_0_EMAIL: "demo@acme.com"
      KEYCLOAK_USERS_0_ROLES_0: "Identity"
      KEYCLOAK_USERS_0_ROLES_1: "Optimize"
      KEYCLOAK_USERS_0_ROLES_2: "Operate"
      KEYCLOAK_USERS_0_ROLES_3: "Tasklist"
      KEYCLOAK_USERS_0_ROLES_4: "Web Modeler"
      KEYCLOAK_CLIENTS_0_NAME: zeebe
      KEYCLOAK_CLIENTS_0_ID: ${ZEEBE_CLIENT_ID}
      KEYCLOAK_CLIENTS_0_SECRET: ${ZEEBE_CLIENT_SECRET}
      KEYCLOAK_CLIENTS_0_TYPE: M2M
      KEYCLOAK_CLIENTS_0_PERMISSIONS_0_RESOURCE_SERVER_ID: zeebe-api
      KEYCLOAK_CLIENTS_0_PERMISSIONS_0_DEFINITION: write:*
      RESOURCE_PERMISSIONS_ENABLED: ${RESOURCE_AUTHORIZATIONS_ENABLED}
    healthcheck:
      test: [ "CMD", "wget", "-q", "--tries=1", "--spider", "http://localhost:8082/actuator/health" ]
      interval: 5s
      timeout: 15s
      retries: 30
      start_period: 60s
    restart: on-failure
    volumes:
      - keycloak-theme:/app/keycloak-theme
    networks:
      - camunda-platform
      - identity-network
    depends_on:
      keycloak:
        condition: service_healthy

  postgres: # https://hub.docker.com/_/postgres
    container_name: postgres
    image: postgres:${POSTGRES_VERSION}
    environment:
      POSTGRES_DB: bitnami_keycloak
      POSTGRES_USER: bn_keycloak
      POSTGRES_PASSWORD: "#3]O?4RGj)DE7Z!9SA5"
    restart: on-failure
    healthcheck:
      test: [ "CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}" ]
      interval: 10s
      timeout: 5s
      retries: 5
    volumes:
      - postgres:/var/lib/postgresql/data
    networks:
      - identity-network

  keycloak: # https://hub.docker.com/r/bitnami/keycloak
    container_name: keycloak
    image: bitnami/keycloak:${KEYCLOAK_SERVER_VERSION}
    volumes:
      - keycloak-theme:/opt/bitnami/keycloak/themes/identity
    ports:
      - "18080:8080"
    environment:
      KEYCLOAK_HTTP_RELATIVE_PATH: /auth
      KEYCLOAK_DATABASE_HOST: postgres
      KEYCLOAK_DATABASE_PASSWORD: "#3]O?4RGj)DE7Z!9SA5"
      KEYCLOAK_ADMIN_USER: admin
      KEYCLOAK_ADMIN_PASSWORD: admin
    restart: on-failure
    healthcheck:
      test: [ "CMD", "curl", "-f", "http://localhost:8080/auth" ]
      interval: 30s
      timeout: 15s
      retries: 5
      start_period: 30s
    networks:
      - camunda-platform
      - identity-network
    depends_on:
      - postgres

  elasticsearch: # https://hub.docker.com/_/elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:${ELASTIC_VERSION}
    container_name: elasticsearch
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      - bootstrap.memory_lock=true
      - discovery.type=single-node
      - xpack.security.enabled=false
      # allow running with low disk space
      - cluster.routing.allocation.disk.threshold_enabled=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    restart: always
    healthcheck:
      test: [ "CMD-SHELL", "curl -f http://localhost:9200/_cat/health | grep -q green" ]
      interval: 30s
      timeout: 5s
      retries: 3
    volumes:
      - elastic:/usr/share/elasticsearch/data
    networks:
      - camunda-platform

  kibana:
    image: docker.elastic.co/kibana/kibana:${ELASTIC_VERSION}
    container_name: kibana
    ports:
      - 5601:5601
    volumes:
      - kibana:/usr/share/kibana/data
    networks:
      - camunda-platform
    depends_on:
      - elasticsearch
    profiles:
      - kibana

volumes:
  zeebe:
  elastic:
  postgres:
  keycloak-theme:
  kibana:

networks:
  # Note there are two bridge networks: One for Camunda Platform and one for Identity.
  # Identity and Keycloak are part of both as they need to be accessible by platform components.
  camunda-platform:
  identity-network:

nathan.loding · October 16, 2023, 8:34pm

Hi @savi003 - you’re right, there’s one more environment variable that needs to be set and I just learned about it today! As I mentioned earlier we have lots of changes coming to our documentation around 8.3 and multi-tenancy, so this will be included in a future (very soon!) release of the docs.

You also need to set ZEEBE_GATEWAY_SECURITY_AUTHENTICATION_MODE to identity in the Zeebe Gateway configuration. See the docs here. Multi-tenancy has a dependency on Identity for authorization, so the Gateway needs to be set to use Identity.

# .env
ZEEBE_AUTHENTICATION_MODE=identity

# docker-compose.yml
services:
  zeebe:
    # ...
    environment:
      - ZEEBE_BROKER_GATEWAY_SECURITY_AUTHENTICATION_MODE=${ZEEBE_AUTHENTICATION_MODE}

savi003 · October 17, 2023, 1:58am

@nathan.loding , now zeebe is working, but I’m getting errors in another services. I will wait until the full documentation is released. Thankyou for your help.

nathan.loding · October 18, 2023, 2:35pm

What errors are you getting? Much of the docs have now been released!

savi003 · October 18, 2023, 4:04pm

in operate

2023-10-18 21:28:52 Caused by: java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Expected Identity to provide authorized tenants, see cause for details
2023-10-18 21:28:52     at java.util.concurrent.CompletableFuture.reportGet(Unknown Source) ~[?:?]
2023-10-18 21:28:52     at java.util.concurrent.CompletableFuture.get(Unknown Source) ~[?:?]
2023-10-18 21:28:52     at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:52) ~[zeebe-client-java-8.3.0.jar!/:8.3.0]
2023-10-18 21:28:52     ... 37 more
2023-10-18 21:28:52 Caused by: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Expected Identity to provide authorized tenants, see cause for details
2023-10-18 21:28:52     at io.grpc.Status.asRuntimeException(Status.java:537) ~[grpc-api-1.58.0.jar!/:1.58.0]
2023-10-18 21:28:52     at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481) ~[grpc-stub-1.58.0.jar!/:1.58.0]
2023-10-18 21:28:52     at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:567) ~[grpc-core-1.58.0.jar!/:1.58.0]
2023-10-18 21:28:52     at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:71) ~[grpc-core-1.58.0.jar!/:1.58.0]
2023-10-18 21:28:52     at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:735) ~[grpc-core-1.58.0.jar!/:1.58.0]
2023-10-18 21:28:52     at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:716) ~[grpc-core-1.58.0.jar!/:1.58.0]
2023-10-18 21:28:52     at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.58.0.jar!/:1.58.0]
2023-10-18 21:28:52     at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[grpc-core-1.58.0.jar!/:1.58.0]
2023-10-18 21:28:52     at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
2023-10-18 21:28:52     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
2023-10-18 21:28:52     at java.lang.Thread.run(Unknown Source) ~[?:?]

in tasklist

2023-10-18 21:34:58 
2023-10-18 21:34:58   _______ ___ ___ ___   _____ _   ___ _  ___    ___ ___ _____
2023-10-18 21:34:58  |_  / __| __| _ ) __| |_   _/_\ / __| |/ / |  |_ _/ __|_   _|
2023-10-18 21:34:58   / /| _|| _|| _ \ _|    | |/ _ \\__ \ ' <| |__ | |\__ \ | |
2023-10-18 21:34:58  /___|___|___|___/___|   |_/_/ \_\___/_|\_\____|___|___/ |_|
2023-10-18 21:34:58 
2023-10-18 21:34:58                                                                             
2023-10-18 21:34:58                                    8.3.0
2023-10-18 21:34:58                                                       
2023-10-18 21:34:58 
2023-10-18 21:34:58 2023-10-18 16:04:58.775  INFO 1 --- [kground-preinit] o.h.v.i.u.Version                        : HV000001: Hibernate Validator 8.0.1.Final
2023-10-18 21:34:59 2023-10-18 16:04:58.997  INFO 1 --- [           main] i.c.t.Application                        : Starting Application using Java 17.0.8.1 with PID 1 (/app/classes started by tasklist in /)
2023-10-18 21:34:59 2023-10-18 16:04:59.001  INFO 1 --- [           main] i.c.t.Application                        : The following 1 profile is active: "identity-auth"
2023-10-18 21:35:01 2023-10-18 16:05:01.349  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'io.camunda.tasklist.webapp.security.CustomMethodSecurityExpressionHandler' of type [io.camunda.tasklist.webapp.security.CustomMethodSecurityExpressionHandler] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2023-10-18 21:35:01 2023-10-18 16:05:01.351  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'io.camunda.tasklist.webapp.security.MethodSecurityConfig' of type [io.camunda.tasklist.webapp.security.MethodSecurityConfig$$SpringCGLIB$$0] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2023-10-18 21:35:01 2023-10-18 16:05:01.755  INFO 1 --- [           main] o.s.b.w.e.t.TomcatWebServer              : Tomcat initialized with port(s): 8080 (http)
2023-10-18 21:35:01 2023-10-18 16:05:01.773  INFO 1 --- [           main] o.a.c.h.Http11NioProtocol                : Initializing ProtocolHandler ["http-nio-8080"]
2023-10-18 21:35:01 2023-10-18 16:05:01.777  INFO 1 --- [           main] o.a.c.c.StandardService                  : Starting service [Tomcat]
2023-10-18 21:35:01 2023-10-18 16:05:01.779  INFO 1 --- [           main] o.a.c.c.StandardEngine                   : Starting Servlet engine: [Apache Tomcat/10.1.12]
2023-10-18 21:35:01 2023-10-18 16:05:01.902  INFO 1 --- [           main] o.a.c.c.C.[.[.[/]                        : Initializing Spring embedded WebApplicationContext
2023-10-18 21:35:01 2023-10-18 16:05:01.904  INFO 1 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 2840 ms
2023-10-18 21:35:02 2023-10-18 16:05:02.128  WARN 1 --- [           main] i.c.t.e.ElasticsearchConnector           : Username and/or password for are empty. Basic authentication for elasticsearch is not used.
2023-10-18 21:35:02 2023-10-18 16:05:02.621  WARN 1 --- [           main] i.c.t.e.ElasticsearchConnector           : Elasticsearch cluster is not accessible
2023-10-18 21:35:02 2023-10-18 16:05:02.766  INFO 1 --- [           main] i.c.t.z.ZeebeConnector                   : Use plaintext connection to zeebe
2023-10-18 21:35:03 2023-10-18 16:05:03.295  INFO 1 --- [           main] o.e.c.EhcacheManager                     : Cache 'tokenCache' created in EhcacheManager.
2023-10-18 21:35:03 2023-10-18 16:05:03.518  WARN 1 --- [           main] .k.a.t.GraphQLJavaToolsAutoConfiguration : GraphQL introspection query disabled! This puts your server in contravention of the GraphQL specification and expectations of most clients, so use this option with caution
2023-10-18 21:35:04 2023-10-18 16:05:04.824  WARN 1 --- [           main] o.s.b.a.e.EndpointId                     : Endpoint ID 'usage-metrics' contains invalid characters, please migrate to a valid format.
2023-10-18 21:35:04 2023-10-18 16:05:04.862  INFO 1 --- [           main] g.k.a.w.s.GraphQLWebAutoConfiguration    : Enabling cors filter
2023-10-18 21:35:04 2023-10-18 16:05:04.966  INFO 1 --- [           main] i.c.t.WebappModuleConfiguration          : Starting module: webapp
2023-10-18 21:35:05 2023-10-18 16:05:05.016  INFO 1 --- [           main] i.c.t.s.SchemaStartup                    : SchemaStartup started.
2023-10-18 21:35:05 2023-10-18 16:05:05.016  INFO 1 --- [           main] i.c.t.s.SchemaStartup                    : SchemaStartup: validate schema.
2023-10-18 21:35:05 2023-10-18 16:05:05.326  INFO 1 --- [           main] i.c.t.s.SchemaStartup                    : SchemaStartup: schema won't be created, it either already exist, or schema creation is disabled in configuration.
2023-10-18 21:35:05 2023-10-18 16:05:05.326  INFO 1 --- [           main] i.c.t.s.SchemaStartup                    : SchemaStartup: migrate schema.
2023-10-18 21:35:05 2023-10-18 16:05:05.333  INFO 1 --- [           main] i.c.t.s.m.e.ElasticsearchStepsRepository : Read step 8.3.0-2_flownode-instance-default-tenantId.json 
2023-10-18 21:35:05 2023-10-18 16:05:05.419  INFO 1 --- [           main] i.c.t.s.m.e.ElasticsearchStepsRepository : Read step 8.3.0-6_variable-default-tenantId.json 
2023-10-18 21:35:05 2023-10-18 16:05:05.421  INFO 1 --- [           main] i.c.t.s.m.e.ElasticsearchStepsRepository : Read step 1.2.0-1_user_add_role.json 
2023-10-18 21:35:05 2023-10-18 16:05:05.422  INFO 1 --- [           main] i.c.t.s.m.e.ElasticsearchStepsRepository : Read step 1.1.0-1_task-variable_script.json 
2023-10-18 21:35:05 2023-10-18 16:05:05.423  INFO 1 --- [           main] i.c.t.s.m.e.ElasticsearchStepsRepository : Read step 1.4.0-0_user_script.json 
2023-10-18 21:35:05 2023-10-18 16:05:05.424  INFO 1 --- [           main] i.c.t.s.m.e.ElasticsearchStepsRepository : Read step 8.3.0-0_process_form.json 
2023-10-18 21:35:05 2023-10-18 16:05:05.425  INFO 1 --- [           main] i.c.t.s.m.e.ElasticsearchStepsRepository : Read step 1.4.0-1_user_delete_fields_script.json 
2023-10-18 21:35:05 2023-10-18 16:05:05.426  INFO 1 --- [           main] i.c.t.s.m.e.ElasticsearchStepsRepository : Read step 8.2.2-0_task_add_follow_up.json 
2023-10-18 21:35:05 2023-10-18 16:05:05.427  INFO 1 --- [           main] i.c.t.s.m.e.ElasticsearchStepsRepository : Read step 8.3.0-5_process-instance-default-tenantId.json 
2023-10-18 21:35:05 2023-10-18 16:05:05.427  INFO 1 --- [           main] i.c.t.s.m.e.ElasticsearchStepsRepository : Read step 8.3.0-1_metric-default-tenantId.json 
2023-10-18 21:35:05 2023-10-18 16:05:05.428  INFO 1 --- [           main] i.c.t.s.m.e.ElasticsearchStepsRepository : Read step 8.2.0-1_process_add_version.json 
2023-10-18 21:35:05 2023-10-18 16:05:05.429  INFO 1 --- [           main] i.c.t.s.m.e.ElasticsearchStepsRepository : Read step 1.3.0-0_task_clean_assignee.json 
2023-10-18 21:35:05 2023-10-18 16:05:05.430  INFO 1 --- [           main] i.c.t.s.m.e.ElasticsearchStepsRepository : Read step 8.3.0-7_draft-task-variable-default-tenantId.json 
2023-10-18 21:35:05 2023-10-18 16:05:05.430  INFO 1 --- [           main] i.c.t.s.m.e.ElasticsearchStepsRepository : Read step 1.2.0-0_user-remove-role.json 
2023-10-18 21:35:05 2023-10-18 16:05:05.431  INFO 1 --- [           main] i.c.t.s.m.e.ElasticsearchStepsRepository : Read step 8.3.0-8_task-default-tenantId.json 
2023-10-18 21:35:05 2023-10-18 16:05:05.431  INFO 1 --- [           main] i.c.t.s.m.e.ElasticsearchStepsRepository : Read step 8.2.0-0_add_sequence_import.json 
2023-10-18 21:35:05 2023-10-18 16:05:05.432  INFO 1 --- [           main] i.c.t.s.m.e.ElasticsearchStepsRepository : Read step 8.2.3-0_task_candidate_users.json 
2023-10-18 21:35:05 2023-10-18 16:05:05.433  INFO 1 --- [           main] i.c.t.s.m.e.ElasticsearchStepsRepository : Read step 8.3.0-4_process-default-tenantId.json 
2023-10-18 21:35:05 2023-10-18 16:05:05.433  INFO 1 --- [           main] i.c.t.s.m.e.ElasticsearchStepsRepository : Read step 8.2.0-0_process_script.json 
2023-10-18 21:35:05 2023-10-18 16:05:05.434  INFO 1 --- [           main] i.c.t.s.m.e.ElasticsearchStepsRepository : Read step 8.3.0-3_form-default-tenantId.json 
2023-10-18 21:35:05 2023-10-18 16:05:05.434  INFO 1 --- [           main] i.c.t.s.m.e.ElasticsearchStepsRepository : Read step 1.1.0-0_variable_script.json 
2023-10-18 21:35:05 2023-10-18 16:05:05.435  INFO 1 --- [           main] i.c.t.s.m.e.ElasticsearchStepsRepository : Read step 8.2.2-1_task_add_due_.json 
2023-10-18 21:35:05 2023-10-18 16:05:05.435  INFO 1 --- [           main] i.c.t.s.m.e.ElasticsearchStepsRepository : Read step 8.3.0-9_task-variable-tenantId.json 
2023-10-18 21:35:05 2023-10-18 16:05:05.516  WARN 1 --- [           main] o.e.c.RestClient                         : request [POST http://elasticsearch:9200/tasklist-migration-steps-repository-1.1.0_/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-8.8.2-98e1271edf932a480e4262a471281f1ee295ce6b "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
2023-10-18 21:35:05 2023-10-18 16:05:05.557  WARN 1 --- [           main] o.e.c.RestClient                         : request [POST http://elasticsearch:9200/tasklist-migration-steps-repository-1.1.0_/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&scroll=60000ms&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-8.8.2-98e1271edf932a480e4262a471281f1ee295ce6b "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
2023-10-18 21:35:05 2023-10-18 16:05:05.612  INFO 1 --- [           main] i.c.t.s.m.e.ElasticsearchStepsRepository : Add new step ProcessorStep [content={"set": {"field": "bpmnProcessId", "value": ""}}, appliedDate=null, indexName=process, isApplied=false, version=8.2.0, order=0, createdDate=2023-10-18T16:05:05.612531698Z] to repository.
2023-10-18 21:35:05 2023-10-18 16:05:05.701  INFO 1 --- [           main] i.c.t.s.m.e.ElasticsearchStepsRepository : Step ProcessorStep [content={"set": {"field": "bpmnProcessId", "value": ""}}, appliedDate=null, indexName=process, isApplied=false, version=8.2.0, order=0, createdDate=2023-10-18T16:05:05.612531698Z]  saved.
2023-10-18 21:35:05 2023-10-18 16:05:05.737  INFO 1 --- [    migration_1] i.c.t.s.m.e.ElasticSearchMigrator        : Check if index web-session needs to migrate.
2023-10-18 21:35:05 2023-10-18 16:05:05.738  INFO 1 --- [    migration_3] i.c.t.s.m.e.ElasticSearchMigrator        : Check if index form needs to migrate.
2023-10-18 21:35:05 2023-10-18 16:05:05.738  INFO 1 --- [    migration_2] i.c.t.s.m.e.ElasticSearchMigrator        : Check if index metric needs to migrate.
2023-10-18 21:35:05 2023-10-18 16:05:05.738  INFO 1 --- [    migration_4] i.c.t.s.m.e.ElasticSearchMigrator        : Check if index user needs to migrate.
2023-10-18 21:35:05 2023-10-18 16:05:05.739  INFO 1 --- [    migration_5] i.c.t.s.m.e.ElasticSearchMigrator        : Check if index variable needs to migrate.
2023-10-18 21:35:05 2023-10-18 16:05:05.744  INFO 1 --- [    migration_4] i.c.t.s.m.e.ElasticSearchMigrator        : No migration needed for user, no previous indices found.
2023-10-18 21:35:05 2023-10-18 16:05:05.744  INFO 1 --- [    migration_1] i.c.t.s.m.e.ElasticSearchMigrator        : No migration needed for web-session, no previous indices found.
2023-10-18 21:35:05 2023-10-18 16:05:05.745  INFO 1 --- [    migration_4] i.c.t.s.m.e.ElasticSearchMigrator        : Check if index migration-steps-repository needs to migrate.
2023-10-18 21:35:05 2023-10-18 16:05:05.745  INFO 1 --- [    migration_3] i.c.t.s.m.e.ElasticSearchMigrator        : No migration needed for form, no previous indices found.
2023-10-18 21:35:05 2023-10-18 16:05:05.745  INFO 1 --- [    migration_3] i.c.t.s.m.e.ElasticSearchMigrator        : Check if index process needs to migrate.
2023-10-18 21:35:05 2023-10-18 16:05:05.745  INFO 1 --- [    migration_2] i.c.t.s.m.e.ElasticSearchMigrator        : No migration needed for metric, no previous indices found.
2023-10-18 21:35:05 2023-10-18 16:05:05.746  INFO 1 --- [    migration_2] i.c.t.s.m.e.ElasticSearchMigrator        : Check if index import-position needs to migrate.
2023-10-18 21:35:05 2023-10-18 16:05:05.746  INFO 1 --- [    migration_1] i.c.t.s.m.e.ElasticSearchMigrator        : Check if index flownode-instance needs to migrate.
2023-10-18 21:35:05 2023-10-18 16:05:05.747  INFO 1 --- [    migration_5] i.c.t.s.m.e.ElasticSearchMigrator        : No migration needed for variable, no previous indices found.
2023-10-18 21:35:05 2023-10-18 16:05:05.748  INFO 1 --- [    migration_5] i.c.t.s.m.e.ElasticSearchMigrator        : Check if index process-instance needs to migrate.
2023-10-18 21:35:05 2023-10-18 16:05:05.751  INFO 1 --- [    migration_3] i.c.t.s.m.e.ElasticSearchMigrator        : No migration needed for process, no previous indices found.
2023-10-18 21:35:05 2023-10-18 16:05:05.752  INFO 1 --- [    migration_4] i.c.t.s.m.e.ElasticSearchMigrator        : No migration needed for migration-steps-repository, no previous indices found.
2023-10-18 21:35:05 2023-10-18 16:05:05.753  INFO 1 --- [    migration_4] i.c.t.s.m.e.ElasticSearchMigrator        : Check if index task-variable needs to migrate.
2023-10-18 21:35:05 2023-10-18 16:05:05.753  INFO 1 --- [    migration_5] i.c.t.s.m.e.ElasticSearchMigrator        : No migration needed for process-instance, no previous indices found.
2023-10-18 21:35:05 2023-10-18 16:05:05.754  INFO 1 --- [    migration_5] i.c.t.s.m.e.ElasticSearchMigrator        : Check if index draft-task-variable needs to migrate.
2023-10-18 21:35:05 2023-10-18 16:05:05.754  INFO 1 --- [    migration_1] i.c.t.s.m.e.ElasticSearchMigrator        : No migration needed for flownode-instance, no previous indices found.
2023-10-18 21:35:05 2023-10-18 16:05:05.751  INFO 1 --- [    migration_2] i.c.t.s.m.e.ElasticSearchMigrator        : No migration needed for import-position, no previous indices found.
2023-10-18 21:35:05 2023-10-18 16:05:05.752  INFO 1 --- [    migration_3] i.c.t.s.m.e.ElasticSearchMigrator        : Check if index task needs to migrate.
2023-10-18 21:35:05 2023-10-18 16:05:05.763  INFO 1 --- [    migration_3] i.c.t.s.m.e.ElasticSearchMigrator        : No migration needed for task, no previous indices found.
2023-10-18 21:35:05 2023-10-18 16:05:05.763  INFO 1 --- [    migration_4] i.c.t.s.m.e.ElasticSearchMigrator        : No migration needed for task-variable, no previous indices found.
2023-10-18 21:35:05 2023-10-18 16:05:05.768  INFO 1 --- [    migration_5] i.c.t.s.m.e.ElasticSearchMigrator        : No migration needed for draft-task-variable, no previous indices found.
2023-10-18 21:35:05 2023-10-18 16:05:05.768  INFO 1 --- [           main] i.c.t.s.SchemaStartup                    : SchemaStartup: finished.
2023-10-18 21:35:05 2023-10-18 16:05:05.772  INFO 1 --- [           main] i.c.t.ImportModuleConfiguration          : Starting module: importer
2023-10-18 21:35:05 2023-10-18 16:05:05.774  INFO 1 --- [           main] i.c.t.ArchiverModuleConfiguration        : Starting module: archiver
2023-10-18 21:35:05 2023-10-18 16:05:05.808  INFO 1 --- [           main] i.c.t.w.StartupBean                      : INIT: DONE
2023-10-18 21:35:06 2023-10-18 16:05:06.179  INFO 1 --- [           main] i.c.t.z.ImportPositionHolderAbstract     : INIT: Start import position updater...
2023-10-18 21:35:06 2023-10-18 16:05:06.484  WARN 1 --- [-worker-ELG-1-4] i.c.z.c.i.ZeebeCallCredentials           : The request's security level does not guarantee that the credentials will be confidential.
2023-10-18 21:35:06 2023-10-18 16:05:06.632  WARN 1 --- [ault-executor-0] i.c.z.c.i.ZeebeCallCredentials           : The request's security level does not guarantee that the credentials will be confidential.
2023-10-18 21:35:06 2023-10-18 16:05:06.658  WARN 1 --- [ault-executor-0] i.c.z.c.i.ZeebeCallCredentials           : The request's security level does not guarantee that the credentials will be confidential.
2023-10-18 21:35:06 2023-10-18 16:05:06.676  WARN 1 --- [ault-executor-0] i.c.z.c.i.ZeebeCallCredentials           : The request's security level does not guarantee that the credentials will be confidential.
2023-10-18 21:35:06 2023-10-18 16:05:06.697  WARN 1 --- [ault-executor-0] i.c.z.c.i.ZeebeCallCredentials           : The request's security level does not guarantee that the credentials will be confidential.
2023-10-18 21:35:06 2023-10-18 16:05:06.717  WARN 1 --- [ault-executor-0] i.c.z.c.i.ZeebeCallCredentials           : The request's security level does not guarantee that the credentials will be confidential.
2023-10-18 21:35:06 2023-10-18 16:05:06.744  WARN 1 --- [ault-executor-0] i.c.z.c.i.ZeebeCallCredentials           : The request's security level does not guarantee that the credentials will be confidential.
2023-10-18 21:35:06 2023-10-18 16:05:06.763  WARN 1 --- [ault-executor-0] i.c.z.c.i.ZeebeCallCredentials           : The request's security level does not guarantee that the credentials will be confidential.
2023-10-18 21:35:06 2023-10-18 16:05:06.787  WARN 1 --- [ault-executor-0] i.c.z.c.i.ZeebeCallCredentials           : The request's security level does not guarantee that the credentials will be confidential.
2023-10-18 21:35:06 2023-10-18 16:05:06.809  WARN 1 --- [ault-executor-0] i.c.z.c.i.ZeebeCallCredentials           : The request's security level does not guarantee that the credentials will be confidential.
2023-10-18 21:35:06 2023-10-18 16:05:06.830  WARN 1 --- [ault-executor-0] i.c.z.c.i.ZeebeCallCredentials           : The request's security level does not guarantee that the credentials will be confidential.
2023-10-18 21:35:06 2023-10-18 16:05:06.855  WARN 1 --- [ault-executor-0] i.c.z.c.i.ZeebeCallCredentials           : The request's security level does not guarantee that the credentials will be confidential.
2023-10-18 21:35:06 2023-10-18 16:05:06.875  WARN 1 --- [ault-executor-0] i.c.z.c.i.ZeebeCallCredentials           : The request's security level does not guarantee that the credentials will be confidential.
2023-10-18 21:35:06 2023-10-18 16:05:06.896  WARN 1 --- [ault-executor-0] i.c.z.c.i.ZeebeCallCredentials           : The request's security level does not guarantee that the credentials will be confidential.
2023-10-18 21:35:06 2023-10-18 16:05:06.917  WARN 1 --- [ault-executor-0] i.c.z.c.i.ZeebeCallCredentials           : The request's security level does not guarantee that the credentials will be confidential.
2023-10-18 21:35:06 2023-10-18 16:05:06.938  WARN 1 --- [ault-executor-0] i.c.z.c.i.ZeebeCallCredentials           : The request's security level does not guarantee that the credentials will be confidential.
2023-10-18 21:35:06 2023-10-18 16:05:06.965  WARN 1 --- [ault-executor-0] i.c.z.c.i.ZeebeCallCredentials           : The request's security level does not guarantee that the credentials will be confidential.
2023-10-18 21:35:06 2023-10-18 16:05:06.989  WARN 1 --- [ault-executor-0] i.c.z.c.i.ZeebeCallCredentials           : The request's security level does not guarantee that the credentials will be confidential.
2023-10-18 21:35:07 2023-10-18 16:05:07.012  WARN 1 --- [ault-executor-0] i.c.z.c.i.ZeebeCallCredentials           : The request's security level does not guarantee that the credentials will be confidential.
2023-10-18 21:35:07 2023-10-18 16:05:07.032  WARN 1 --- [ault-executor-2] i.c.z.c.i.ZeebeCallCredentials           : The request's security level does not guarantee that the credentials will be confidential.
2023-10-18 21:35:07 2023-10-18 16:05:07.053  WARN 1 --- [ault-executor-2] i.c.z.c.i.ZeebeCallCredentials           : The request's security level does not guarantee that the credentials will be confidential.
2023-10-18 21:35:07 2023-10-18 16:05:07.071  WARN 1 --- [ault-executor-2] i.c.z.c.i.ZeebeCallCredentials           : The request's security level does not guarantee that the credentials will

nathan.loding · October 18, 2023, 9:12pm

The “request’s security level…” warning message is pretty common and not related to multi-tenancy. I do not get that error with Operate when I used the docker-compose.yaml file you provided. Does that error occur on startup, or does it happen when you try to do something within Operate?

savi003 · October 19, 2023, 1:45am

# While the Docker images themselves are supported for production usage,
# this docker-compose.yaml is designed to be used by developers to run
# an environment locally. It is not designed to be used in production.
# We recommend to use Kubernetes in production with our Helm Charts:
# https://docs.camunda.io/docs/self-managed/platform-deployment/kubernetes-helm/
# For local development, we recommend using KIND instead of `docker-compose`:
# https://docs.camunda.io/docs/self-managed/platform-deployment/helm-kubernetes/guides/local-kubernetes-cluster/

# This is a full configuration with Zeebe, Operate, Tasklist, Optimize, Identity, Keycloak, and Elasticsearch
# See docker-compose-core.yml for a lightweight configuration that does not include Optimize, Identity, and Keycloak.

services:

  zeebe: # https://docs.camunda.io/docs/self-managed/platform-deployment/docker/#zeebe
    image: camunda/zeebe:${CAMUNDA_PLATFORM_VERSION}
    container_name: zeebe
    ports:
      - "26500:26500"
      - "9600:9600"
    environment: # https://docs.camunda.io/docs/self-managed/zeebe-deployment/configuration/environment-variables/
      - ZEEBE_BROKER_GATEWAY_SECURITY_AUTHENTICATION_MODE=${ZEEBE_AUTHENTICATION_MODE}
      - ZEEBE_BROKER_GATEWAY_SECURITY_AUTHENTICATION_IDENTITY_ISSUERBACKENDURL=http://keycloak:8080/auth/realms/camunda-platform
      - ZEEBE_BROKER_GATEWAY_SECURITY_AUTHENTICATION_IDENTITY_AUDIENCE=zeebe-api
      - ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_CLASSNAME=io.camunda.zeebe.exporter.ElasticsearchExporter
      - ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_URL=http://elasticsearch:9200
      # default is 1000, see here: https://github.com/camunda/zeebe/blob/cd443f118a5c622b2902589b6d75ea9fde3ca0af/exporters/elasticsearch-exporter/src/main/java/io/camunda/zeebe/exporter/ElasticsearchExporterConfiguration.java#L259
      - ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_BULK_SIZE=1
      # allow running with low disk space
      - ZEEBE_BROKER_DATA_DISKUSAGECOMMANDWATERMARK=0.998
      - ZEEBE_BROKER_DATA_DISKUSAGEREPLICATIONWATERMARK=0.999
      - ZEEBE_GATEWAY_MULTITENANCY_ENABLED=${MULTITENANCY_ENABLED}
      - ZEEBE_BROKER_GATEWAY_MULTITENANCY_ENABLED=${MULTITENANCY_ENABLED}
      - "JAVA_TOOL_OPTIONS=-Xms512m -Xmx512m"
    restart: always
    healthcheck:
      test: [ "CMD-SHELL", "timeout 10s bash -c ':> /dev/tcp/127.0.0.1/9600' || exit 1" ]
      interval: 30s
      timeout: 5s
      retries: 5
      start_period: 30s
    volumes:
      - zeebe:/usr/local/zeebe/data
    networks:
      - camunda-platform
    depends_on:
      - elasticsearch
      - identity

  operate: # https://docs.camunda.io/docs/self-managed/platform-deployment/docker/#operate
    image: camunda/operate:${CAMUNDA_PLATFORM_VERSION}
    container_name: operate
    ports:
      - "8081:8080"
    environment: # https://docs.camunda.io/docs/self-managed/operate-deployment/configuration/
      - CAMUNDA_OPERATE_ZEEBE_GATEWAYADDRESS=zeebe:26500
      - ZEEBE_CLIENT_ID=${ZEEBE_CLIENT_ID}
      - ZEEBE_CLIENT_SECRET=${ZEEBE_CLIENT_SECRET}
      - ZEEBE_TOKEN_AUDIENCE=zeebe-api
      - ZEEBE_AUTHORIZATION_SERVER_URL=http://keycloak:8080/auth/realms/camunda-platform/protocol/openid-connect/token
      - CAMUNDA_OPERATE_ELASTICSEARCH_URL=http://elasticsearch:9200
      - CAMUNDA_OPERATE_ZEEBEELASTICSEARCH_URL=http://elasticsearch:9200
      # For more information regarding configuration with Identity see:
      # https://docs.camunda.io/docs/self-managed/operate-deployment/authentication/#identity
      - SPRING_PROFILES_ACTIVE=identity-auth
      - CAMUNDA_OPERATE_IDENTITY_BASEURL=http://identity:8084
      - CAMUNDA_OPERATE_IDENTITY_ISSUER_URL=http://${HOST}:18080/auth/realms/camunda-platform
      - CAMUNDA_OPERATE_IDENTITY_ISSUER_BACKEND_URL=http://keycloak:8080/auth/realms/camunda-platform
      - CAMUNDA_OPERATE_IDENTITY_CLIENTID=operate
      - CAMUNDA_OPERATE_IDENTITY_CLIENTSECRET=XALaRPl5qwTEItdwCMiPS62nVpKs7dL7
      - CAMUNDA_OPERATE_IDENTITY_AUDIENCE=operate-api
      - SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_ISSUER_URI=http://${HOST}:18080/auth/realms/camunda-platform
      - SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_JWK_SET_URI=http://${HOST}:18080/auth/realms/camunda-platform/protocol/openid-connect/certs
      - CAMUNDA_OPERATE_IDENTITY_RESOURCEPERMISSIONSENABLED=${RESOURCE_AUTHORIZATIONS_ENABLED}
      - camunda.operate.multiTenancy.enabled=${MULTITENANCY_ENABLED}
      - management.endpoints.web.exposure.include=health
      - management.endpoint.health.probes.enabled=true
    healthcheck:
      test: [ "CMD-SHELL", "curl -f http://localhost:8080/actuator/health/readiness" ]
      interval: 30s
      timeout: 1s
      retries: 5
      start_period: 30s
    networks:
      - camunda-platform
    depends_on:
      - zeebe
      - identity
      - elasticsearch

  tasklist: # https://docs.camunda.io/docs/self-managed/platform-deployment/docker/#tasklist
    image: camunda/tasklist:${CAMUNDA_PLATFORM_VERSION}
    container_name: tasklist
    ports:
      - "8082:8080"
    environment: # https://docs.camunda.io/docs/self-managed/tasklist-deployment/configuration/
      - CAMUNDA_TASKLIST_ZEEBE_GATEWAYADDRESS=zeebe:26500
      - ZEEBE_CLIENT_ID=${ZEEBE_CLIENT_ID}
      - ZEEBE_CLIENT_SECRET=${ZEEBE_CLIENT_SECRET}
      - ZEEBE_TOKEN_AUDIENCE=zeebe-api
      - ZEEBE_AUTHORIZATION_SERVER_URL=http://keycloak:8080/auth/realms/camunda-platform/protocol/openid-connect/token
      - CAMUNDA_TASKLIST_ELASTICSEARCH_URL=http://elasticsearch:9200
      - CAMUNDA_TASKLIST_ZEEBEELASTICSEARCH_URL=http://elasticsearch:9200
      # For more information regarding configuration with Identity see:
      # https://docs.camunda.io/docs/self-managed/tasklist-deployment/authentication/#identity
      - SPRING_PROFILES_ACTIVE=identity-auth
      - CAMUNDA_TASKLIST_IDENTITY_BASEURL=http://identity:8084
      - CAMUNDA_TASKLIST_IDENTITY_ISSUER_URL=http://${HOST}:18080/auth/realms/camunda-platform
      - CAMUNDA_TASKLIST_IDENTITY_ISSUER_BACKEND_URL=http://keycloak:8080/auth/realms/camunda-platform
      - CAMUNDA_TASKLIST_IDENTITY_CLIENTID=tasklist
      - CAMUNDA_TASKLIST_IDENTITY_CLIENTSECRET=XALaRPl5qwTEItdwCMiPS62nVpKs7dL7
      - CAMUNDA_TASKLIST_IDENTITY_AUDIENCE=tasklist-api
      - SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_ISSUER_URI=http://${HOST}:18080/auth/realms/camunda-platform
      - SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_JWK_SET_URI=http://${HOST}:18080/auth/realms/camunda-platform/protocol/openid-connect/certs
      - CAMUNDA_TASKLIST_IDENTITY_RESOURCE_PERMISSIONS_ENABLED=${RESOURCE_AUTHORIZATIONS_ENABLED}
      - CAMUNDA_TASKLIST_MULTITENANCY_ENABLED=${MULTITENANCY_ENABLED}
      - management.endpoints.web.exposure.include=health
      - management.endpoint.health.probes.enabled=true
    healthcheck:
      test: [ "CMD-SHELL", "curl -f http://localhost:8080/actuator/health/readiness" ]
      interval: 30s
      timeout: 1s
      retries: 5
      start_period: 30s
    networks:
      - camunda-platform
    depends_on:
      zeebe:
        condition: service_started
      elasticsearch:
        condition: service_healthy
      identity:
        condition: service_healthy

  connectors: # https://docs.camunda.io/docs/components/integration-framework/connectors/out-of-the-box-connectors/available-connectors-overview/
    image: camunda/connectors-bundle:${CAMUNDA_CONNECTORS_VERSION}
    container_name: connectors
    ports:
      - "8085:8080"
    environment:
      - ZEEBE_CLIENT_BROKER_GATEWAY-ADDRESS=zeebe:26500
      - ZEEBE_CLIENT_SECURITY_PLAINTEXT=true
      - ZEEBE_CLIENT_ID=${ZEEBE_CLIENT_ID}
      - ZEEBE_CLIENT_SECRET=${ZEEBE_CLIENT_SECRET}
      - ZEEBE_TOKEN_AUDIENCE=zeebe-api
      - ZEEBE_AUTHORIZATION_SERVER_URL=http://keycloak:8080/auth/realms/camunda-platform/protocol/openid-connect/token
      - CAMUNDA_OPERATE_CLIENT_KEYCLOAK-URL=http://keycloak:8080
      - CAMUNDA_OPERATE_CLIENT_CLIENT-ID=connectors
      - CAMUNDA_OPERATE_CLIENT_CLIENT-SECRET=XALaRPl5qwTEItdwCMiPS62nVpKs7dL7
      - CAMUNDA_OPERATE_CLIENT_KEYCLOAK-REALM=camunda-platform
      - CAMUNDA_OPERATE_CLIENT_URL=http://operate:8080
      - management.endpoints.web.exposure.include=health
      - management.endpoint.health.probes.enabled=true
    env_file: connector-secrets.txt
    healthcheck:
      test: [ "CMD-SHELL", "curl -f http://localhost:8080/actuator/health/readiness" ]
      interval: 30s
      timeout: 1s
      retries: 5
      start_period: 30s
    networks:
      - camunda-platform
    depends_on:
      - zeebe
      - operate
      - identity

  optimize: # https://docs.camunda.io/docs/self-managed/platform-deployment/docker/#optimize
    image: camunda/optimize:${CAMUNDA_OPTIMIZE_VERSION}
    container_name: optimize
    ports:
      - "8083:8090"
    environment: # https://docs.camunda.io/docs/self-managed/optimize-deployment/setup/installation/#available-environment-variables
      - OPTIMIZE_ELASTICSEARCH_HOST=elasticsearch
      - OPTIMIZE_ELASTICSEARCH_HTTP_PORT=9200
      - SPRING_PROFILES_ACTIVE=ccsm
      - CAMUNDA_OPTIMIZE_ZEEBE_ENABLED=true
      - CAMUNDA_OPTIMIZE_ENTERPRISE=false
      - CAMUNDA_OPTIMIZE_IDENTITY_ISSUER_URL=http://${HOST}:18080/auth/realms/camunda-platform
      - CAMUNDA_OPTIMIZE_IDENTITY_ISSUER_BACKEND_URL=http://keycloak:8080/auth/realms/camunda-platform
      - CAMUNDA_OPTIMIZE_IDENTITY_CLIENTID=optimize
      - CAMUNDA_OPTIMIZE_IDENTITY_CLIENTSECRET=XALaRPl5qwTEItdwCMiPS62nVpKs7dL7
      - CAMUNDA_OPTIMIZE_IDENTITY_AUDIENCE=optimize-api
      - CAMUNDA_OPTIMIZE_SECURITY_AUTH_COOKIE_SAME_SITE_ENABLED=false
      - CAMUNDA_OPTIMIZE_UI_LOGOUT_HIDDEN=true
      - CAMUNDA_OPTIMIZE_MULTITENANCY_ENABLED=${MULTITENANCY_ENABLED}
      - management.endpoints.web.exposure.include=health
      - management.endpoint.health.probes.enabled=true
    healthcheck:
      test: [ "CMD-SHELL", "curl -f http://localhost:8090/api/readyz" ]
      interval: 30s
      timeout: 1s
      retries: 5
      start_period: 30s
    volumes:
      - "./.optimize/environment-config.yaml:/optimize/config/environment-config.yaml"
    restart: on-failure
    networks:
      - camunda-platform
    depends_on:
      - identity
      - elasticsearch

  identity: # https://docs.camunda.io/docs/self-managed/platform-deployment/docker/#identity
    container_name: identity
    image: camunda/identity:${CAMUNDA_PLATFORM_VERSION}
    ports:
      - "8084:8084"
    environment: # https://docs.camunda.io/docs/self-managed/identity/deployment/configuration-variables/
      SERVER_PORT: 8084
      IDENTITY_RETRY_DELAY_SECONDS: 30
      KEYCLOAK_URL: http://keycloak:8080/auth
      IDENTITY_AUTH_PROVIDER_BACKEND_URL: http://keycloak:8080/auth/realms/camunda-platform
      IDENTITY_DATABASE_HOST: postgres
      IDENTITY_DATABASE_PORT: 5432
      IDENTITY_DATABASE_NAME: bitnami_keycloak
      IDENTITY_DATABASE_USERNAME: bn_keycloak
      IDENTITY_DATABASE_PASSWORD: "#3]O?4RGj)DE7Z!9SA5"
      KEYCLOAK_INIT_OPERATE_SECRET: XALaRPl5qwTEItdwCMiPS62nVpKs7dL7
      KEYCLOAK_INIT_OPERATE_ROOT_URL: http://${HOST}:8081
      KEYCLOAK_INIT_TASKLIST_SECRET: XALaRPl5qwTEItdwCMiPS62nVpKs7dL7
      KEYCLOAK_INIT_TASKLIST_ROOT_URL: http://${HOST}:8082
      KEYCLOAK_INIT_OPTIMIZE_SECRET: XALaRPl5qwTEItdwCMiPS62nVpKs7dL7
      KEYCLOAK_INIT_OPTIMIZE_ROOT_URL: http://${HOST}:8083
      KEYCLOAK_INIT_WEBMODELER_ROOT_URL: http://${HOST}:8070
      KEYCLOAK_INIT_CONNECTORS_SECRET: XALaRPl5qwTEItdwCMiPS62nVpKs7dL7
      KEYCLOAK_INIT_CONNECTORS_ROOT_URL: http://${HOST}:8085
      KEYCLOAK_INIT_ZEEBE_NAME: zeebe
      KEYCLOAK_USERS_0_USERNAME: "demo"
      KEYCLOAK_USERS_0_PASSWORD: "demo"
      KEYCLOAK_USERS_0_FIRST_NAME: "demo"
      KEYCLOAK_USERS_0_EMAIL: "demo@acme.com"
      KEYCLOAK_USERS_0_ROLES_0: "Identity"
      KEYCLOAK_USERS_0_ROLES_1: "Optimize"
      KEYCLOAK_USERS_0_ROLES_2: "Operate"
      KEYCLOAK_USERS_0_ROLES_3: "Tasklist"
      KEYCLOAK_USERS_0_ROLES_4: "Web Modeler"
      KEYCLOAK_CLIENTS_0_NAME: zeebe
      KEYCLOAK_CLIENTS_0_ID: ${ZEEBE_CLIENT_ID}
      KEYCLOAK_CLIENTS_0_SECRET: ${ZEEBE_CLIENT_SECRET}
      KEYCLOAK_CLIENTS_0_TYPE: M2M
      KEYCLOAK_CLIENTS_0_PERMISSIONS_0_RESOURCE_SERVER_ID: zeebe-api
      KEYCLOAK_CLIENTS_0_PERMISSIONS_0_DEFINITION: write:*
      RESOURCE_PERMISSIONS_ENABLED: ${RESOURCE_AUTHORIZATIONS_ENABLED}
      MULTITENANCY_ENABLED: ${MULTITENANCY_ENABLED}
    healthcheck:
      test: [ "CMD", "wget", "-q", "--tries=1", "--spider", "http://localhost:8082/actuator/health" ]
      interval: 5s
      timeout: 15s
      retries: 30
      start_period: 60s
    restart: on-failure
    volumes:
      - keycloak-theme:/app/keycloak-theme
    networks:
      - camunda-platform
      - identity-network
    depends_on:
      keycloak:
        condition: service_healthy

  postgres: # https://hub.docker.com/_/postgres
    container_name: postgres
    image: postgres:${POSTGRES_VERSION}
    environment:
      POSTGRES_DB: bitnami_keycloak
      POSTGRES_USER: bn_keycloak
      POSTGRES_PASSWORD: "#3]O?4RGj)DE7Z!9SA5"
    restart: on-failure
    healthcheck:
      test: [ "CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}" ]
      interval: 10s
      timeout: 5s
      retries: 5
    volumes:
      - postgres:/var/lib/postgresql/data
    networks:
      - identity-network

  keycloak: # https://hub.docker.com/r/bitnami/keycloak
    container_name: keycloak
    image: bitnami/keycloak:${KEYCLOAK_SERVER_VERSION}
    volumes:
      - keycloak-theme:/opt/bitnami/keycloak/themes/identity
    ports:
      - "18080:8080"
    environment:
      KEYCLOAK_HTTP_RELATIVE_PATH: /auth
      KEYCLOAK_DATABASE_HOST: postgres
      KEYCLOAK_DATABASE_PASSWORD: "#3]O?4RGj)DE7Z!9SA5"
      KEYCLOAK_ADMIN_USER: admin
      KEYCLOAK_ADMIN_PASSWORD: admin
    restart: on-failure
    healthcheck:
      test: [ "CMD", "curl", "-f", "http://localhost:8080/auth" ]
      interval: 30s
      timeout: 15s
      retries: 5
      start_period: 30s
    networks:
      - camunda-platform
      - identity-network
    depends_on:
      - postgres

  elasticsearch: # https://hub.docker.com/_/elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:${ELASTIC_VERSION}
    container_name: elasticsearch
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      - bootstrap.memory_lock=true
      - discovery.type=single-node
      - xpack.security.enabled=false
      # allow running with low disk space
      - cluster.routing.allocation.disk.threshold_enabled=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    restart: always
    healthcheck:
      test: [ "CMD-SHELL", "curl -f http://localhost:9200/_cat/health | grep -q green" ]
      interval: 30s
      timeout: 5s
      retries: 3
    volumes:
      - elastic:/usr/share/elasticsearch/data
    networks:
      - camunda-platform

  kibana:
    image: docker.elastic.co/kibana/kibana:${ELASTIC_VERSION}
    container_name: kibana
    ports:
      - 5601:5601
    volumes:
      - kibana:/usr/share/kibana/data
    networks:
      - camunda-platform
    depends_on:
      - elasticsearch
    profiles:
      - kibana

volumes:
  zeebe:
  elastic:
  postgres:
  keycloak-theme:
  kibana:

networks:
  # Note there are two bridge networks: One for Camunda Platform and one for Identity.
  # Identity and Keycloak are part of both as they need to be accessible by platform components.
  camunda-platform:
  identity-network:

this is my docker-compose.yaml file
i have created an environment variable in .env file, so that i can manage it easily

MULTITENANCY_ENABLED=true

on startup, operate is showing error and then shutting down, the error is

2023-10-19 07:16:28 2023-10-19 01:46:28.703  INFO 7 --- [           main] i.c.o.z.PartitionHolder                  : Partition ids can't be fetched from Zeebe. Try next round (60).
2023-10-19 07:16:29 2023-10-19 01:46:29.703  WARN 7 --- [           main] i.c.z.c.i.ZeebeCallCredentials           : The request's security level does not guarantee that the credentials will be confidential.
2023-10-19 07:16:29 2023-10-19 01:46:29.709 ERROR 7 --- [ault-executor-0] i.c.z.c.i.o.OAuthCredentialsProvider     : Failed while fetching credentials: 
2023-10-19 07:16:29 
2023-10-19 07:16:29 java.nio.file.AccessDeniedException: /usr/local/operate/.camunda
2023-10-19 07:16:29     at sun.nio.fs.UnixException.translateToIOException(Unknown Source) ~[?:?]
2023-10-19 07:16:29     at sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[?:?]
2023-10-19 07:16:29     at sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[?:?]
2023-10-19 07:16:29     at sun.nio.fs.UnixFileSystemProvider.createDirectory(Unknown Source) ~[?:?]
2023-10-19 07:16:29     at java.nio.file.Files.createDirectory(Unknown Source) ~[?:?]
2023-10-19 07:16:29     at java.nio.file.Files.createAndCheckIsDirectory(Unknown Source) ~[?:?]
2023-10-19 07:16:29     at java.nio.file.Files.createDirectories(Unknown Source) ~[?:?]
2023-10-19 07:16:29     at io.camunda.zeebe.client.impl.oauth.OAuthCredentialsCache.ensureCacheFileExists(OAuthCredentialsCache.java:147) ~[zeebe-client-java-8.3.0.jar!/:8.3.0]
2023-10-19 07:16:29     at io.camunda.zeebe.client.impl.oauth.OAuthCredentialsCache.writeCache(OAuthCredentialsCache.java:74) ~[zeebe-client-java-8.3.0.jar!/:8.3.0]
2023-10-19 07:16:29     at io.camunda.zeebe.client.impl.oauth.OAuthCredentialsProvider.lambda$shouldRetryRequest$0(OAuthCredentialsProvider.java:110) ~[zeebe-client-java-8.3.0.jar!/:8.3.0]
2023-10-19 07:16:29     at io.camunda.zeebe.client.impl.oauth.OAuthCredentialsCache.withCache(OAuthCredentialsCache.java:111) ~[zeebe-client-java-8.3.0.jar!/:8.3.0]
2023-10-19 07:16:29     at io.camunda.zeebe.client.impl.oauth.OAuthCredentialsProvider.shouldRetryRequest(OAuthCredentialsProvider.java:106) ~[zeebe-client-java-8.3.0.jar!/:8.3.0]
2023-10-19 07:16:29     at io.camunda.zeebe.client.impl.RetriableClientFutureImpl.onError(RetriableClientFutureImpl.java:48) [zeebe-client-java-8.3.0.jar!/:8.3.0]
2023-10-19 07:16:29     at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481) [grpc-stub-1.58.0.jar!/:1.58.0]
2023-10-19 07:16:29     at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:567) [grpc-core-1.58.0.jar!/:1.58.0]
2023-10-19 07:16:29     at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:71) [grpc-core-1.58.0.jar!/:1.58.0]
2023-10-19 07:16:29     at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:735) [grpc-core-1.58.0.jar!/:1.58.0]
2023-10-19 07:16:29     at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:716) [grpc-core-1.58.0.jar!/:1.58.0]
2023-10-19 07:16:29     at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.58.0.jar!/:1.58.0]
2023-10-19 07:16:29     at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) [grpc-core-1.58.0.jar!/:1.58.0]
2023-10-19 07:16:29     at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
2023-10-19 07:16:29     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
2023-10-19 07:16:29     at java.lang.Thread.run(Unknown Source) [?:?]
2023-10-19 07:16:29 
2023-10-19 07:16:29 2023-10-19 01:46:29.710  WARN 7 --- [           main] i.c.o.z.PartitionHolder                  : Error occurred when requesting partition ids from Zeebe client: Expected Identity to provide authorized tenants, see cause for details
2023-10-19 07:16:29 
2023-10-19 07:16:29 io.camunda.zeebe.client.api.command.ClientStatusException: Expected Identity to provide authorized tenants, see cause for details
2023-10-19 07:16:29     at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.transformExecutionException(ZeebeClientFutureImpl.java:116) ~[zeebe-client-java-8.3.0.jar!/:8.3.0]
2023-10-19 07:16:29     at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:54) ~[zeebe-client-java-8.3.0.jar!/:8.3.0]
2023-10-19 07:16:29     at io.camunda.operate.zeebe.PartitionHolder.getPartitionIdsFromZeebe(PartitionHolder.java:102) ~[operate-common-8.3.0.jar!/:?]
2023-10-19 07:16:29     at io.camunda.operate.zeebe.PartitionHolder.getPartitionIdsWithWaitingTimeAndRetries(PartitionHolder.java:65) ~[operate-common-8.3.0.jar!/:?]
2023-10-19 07:16:29     at io.camunda.operate.zeebe.PartitionHolder.getPartitionIds(PartitionHolder.java:50) ~[operate-common-8.3.0.jar!/:?]
2023-10-19 07:16:29     at io.camunda.operate.zeebeimport.RecordsReaderHolder.getAllRecordsReaders(RecordsReaderHolder.java:51) ~[operate-importer-8.3.0.jar!/:?]
2023-10-19 07:16:29     at io.camunda.operate.zeebeimport.ZeebeImporter.scheduleReaders(ZeebeImporter.java:55) ~[operate-importer-8.3.0.jar!/:?]
2023-10-19 07:16:29     at io.camunda.operate.zeebeimport.ZeebeImporter.startImportingData(ZeebeImporter.java:48) ~[operate-importer-8.3.0.jar!/:?]
2023-10-19 07:16:29     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
2023-10-19 07:16:29     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
2023-10-19 07:16:29     at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
2023-10-19 07:16:29     at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
2023-10-19 07:16:29     at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMethod.invoke(InitDestroyAnnotationBeanPostProcessor.java:457) ~[spring-beans-6.0.12.jar!/:6.0.12]
2023-10-19 07:16:29     at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:401) ~[spring-beans-6.0.12.jar!/:6.0.12]
2023-10-19 07:16:29     at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:219) ~[spring-beans-6.0.12.jar!/:6.0.12]
2023-10-19 07:16:29     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:419) ~[spring-beans-6.0.12.jar!/:6.0.12]
2023-10-19 07:16:29     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1762) ~[spring-beans-6.0.12.jar!/:6.0.12]
2023-10-19 07:16:29     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:598) ~[spring-beans-6.0.12.jar!/:6.0.12]
2023-10-19 07:16:29     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520) ~[spring-beans-6.0.12.jar!/:6.0.12]
2023-10-19 07:16:29     at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:325) ~[spring-beans-6.0.12.jar!/:6.0.12]
2023-10-19 07:16:29     at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) [spring-beans-6.0.12.jar!/:6.0.12]
2023-10-19 07:16:29     at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:323) [spring-beans-6.0.12.jar!/:6.0.12]
2023-10-19 07:16:29     at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) [spring-beans-6.0.12.jar!/:6.0.12]
2023-10-19 07:16:29     at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:973) [spring-beans-6.0.12.jar!/:6.0.12]
2023-10-19 07:16:29     at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:942) [spring-context-6.0.12.jar!/:6.0.12]
2023-10-19 07:16:29     at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:608) [spring-context-6.0.12.jar!/:6.0.12]
2023-10-19 07:16:29     at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146) [spring-boot-3.1.4.jar!/:3.1.4]
2023-10-19 07:16:29     at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:737) [spring-boot-3.1.4.jar!/:3.1.4]
2023-10-19 07:16:29     at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:439) [spring-boot-3.1.4.jar!/:3.1.4]
2023-10-19 07:16:29     at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) [spring-boot-3.1.4.jar!/:3.1.4]
2023-10-19 07:16:29     at io.camunda.operate.Application.main(Application.java:67) [classes!/:?]
2023-10-19 07:16:29     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
2023-10-19 07:16:29     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
2023-10-19 07:16:29     at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
2023-10-19 07:16:29     at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
2023-10-19 07:16:29     at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) [operate-webapp-8.3.0-exec.jar:?]
2023-10-19 07:16:29     at org.springframework.boot.loader.Launcher.launch(Launcher.java:95) [operate-webapp-8.3.0-exec.jar:?]
2023-10-19 07:16:29     at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) [operate-webapp-8.3.0-exec.jar:?]
2023-10-19 07:16:29     at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65) [operate-webapp-8.3.0-exec.jar:?]
2023-10-19 07:16:29 Caused by: java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Expected Identity to provide authorized tenants, see cause for details
2023-10-19 07:16:29     at java.util.concurrent.CompletableFuture.reportGet(Unknown Source) ~[?:?]
2023-10-19 07:16:29     at java.util.concurrent.CompletableFuture.get(Unknown Source) ~[?:?]
2023-10-19 07:16:29     at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:52) ~[zeebe-client-java-8.3.0.jar!/:8.3.0]
2023-10-19 07:16:29     ... 37 more
2023-10-19 07:16:29 Caused by: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Expected Identity to provide authorized tenants, see cause for details
2023-10-19 07:16:29     at io.grpc.Status.asRuntimeException(Status.java:537) ~[grpc-api-1.58.0.jar!/:1.58.0]
2023-10-19 07:16:29     at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481) ~[grpc-stub-1.58.0.jar!/:1.58.0]
2023-10-19 07:16:29     at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:567) ~[grpc-core-1.58.0.jar!/:1.58.0]
2023-10-19 07:16:29     at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:71) ~[grpc-core-1.58.0.jar!/:1.58.0]
2023-10-19 07:16:29     at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:735) ~[grpc-core-1.58.0.jar!/:1.58.0]
2023-10-19 07:16:29     at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:716) ~[grpc-core-1.58.0.jar!/:1.58.0]
2023-10-19 07:16:29     at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.58.0.jar!/:1.58.0]
2023-10-19 07:16:29     at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[grpc-core-1.58.0.jar!/:1.58.0]
2023-10-19 07:16:29     at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
2023-10-19 07:16:29     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
2023-10-19 07:16:29     at java.lang.Thread.run(Unknown Source) ~[?:?]
2023-10-19 07:16:29 
2023-10-19 07:16:29 2023-10-19 01:46:29.712  INFO 7 --- [           main] i.c.o.z.PartitionHolder                  : Partition ids can't be fetched from Zeebe. Return empty partition ids list.
2023-10-19 07:16:29 2023-10-19 01:46:29.712 ERROR 7 --- [           main] i.c.o.z.PartitionHolder                  : Operate needs an established connection to Zeebe in order to retrieve Zeebe Partitions. Shutting down...
2023-10-19 07:16:29 2023-10-19 01:46:29.727  INFO 7 --- [           main] i.c.o.w.StartupBean                      : Shutdown elasticsearch clients.
2023-10-19 07:16:29 2023-10-19 01:46:29.735  INFO 7 --- [           main] i.c.o.w.z.o.OperationExecutor            : Shutdown OperationExecutor

jgeek1 · October 19, 2023, 9:30am

Multi-tenancy has a dependency on Identity for authorization, so the Gateway needs to be set to use Identity.

@nathan.loding - one related question - Since Identity is available only with proprietary license, does this mean multi-tenancy feature can be enabled only in enterprise deployments?