Operate shows no processes in self managed environment

pacman8 · April 10, 2024, 2:28pm

Hi Camunda team
Im trying to get to run a camunda instance in a plain docker remote environment. Everything seems to work fine (zeebe, tasklist, optimizer, webhooks, jobworkers, …), but operate shows me no running process instances. Few months agoe I was able to run an environment on 8.3.0-alpha1 with a working operate. But on 8.3.3, 8.4.6 or 8.5.0 I had no chance to get operate to work normaly.

Operate log shows me no error. The docker containers are running on a remote centos 7 server. Modifications I made on your provided docker-composer.yaml was only changing hostnames in order not to get forwarded to localhost.

Thanks in advance

Praveen_Kumar_Reddy · April 10, 2024, 4:14pm

Hi @pacman8, Welcome to camunda community forum

Please post any screenshot of the issue or modification done to the yaml

pacman8 · April 11, 2024, 8:12am

Hi @Praveen_Kumar_Reddy
This is what optimizer shows to me:

And this is what operate shows:

This is my docker-composer.yaml:

services:

zeebe: # Docker | Camunda 8 Docs
image: camunda/zeebe:${CAMUNDA_PLATFORM_VERSION}
container_name: zeebe
ports:
- “26500:26500”
- “9600:9600”
environment: # Environment variables | Camunda 8 Docs
- ZEEBE_BROKER_GATEWAY_SECURITY_AUTHENTICATION_MODE=${ZEEBE_AUTHENTICATION_MODE}
- ZEEBE_BROKER_GATEWAY_SECURITY_AUTHENTICATION_IDENTITY_ISSUERBACKENDURL=http://${HOST}:18080/auth/realms/camunda-platform
- ZEEBE_BROKER_GATEWAY_SECURITY_AUTHENTICATION_IDENTITY_AUDIENCE=zeebe-api
- ZEEBE_BROKER_GATEWAY_SECURITY_AUTHENTICATION_IDENTITY_BASEURL=http://${HOST}:8084
- ZEEBE_BROKER_GATEWAY_MULTITENANCY_ENABLED=${MULTI_TENANCY_ENABLED}
- ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_CLASSNAME=io.camunda.zeebe.exporter.ElasticsearchExporter
- ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_URL=http://elasticsearch:9200
# default is 1000, see here: https://github.com/camunda/zeebe/blob/d058cdbf88ce8a613747fd2c44c8f69abe2ca4fb/exporters/elasticsearch-exporter/src/main/java/io/camunda/zeebe/exporter/ElasticsearchExporterConfiguration.java#L259
- ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_BULK_SIZE=1
# allow running with low disk space
- ZEEBE_BROKER_DATA_DISKUSAGECOMMANDWATERMARK=0.998
- ZEEBE_BROKER_DATA_DISKUSAGEREPLICATIONWATERMARK=0.999
- “JAVA_TOOL_OPTIONS=-Xms512m -Xmx512m”
restart: always
healthcheck:
test: [ “CMD-SHELL”, “timeout 10s bash -c ‘:> /dev/tcp/127.0.0.1/9600’ || exit 1” ]
interval: 30s
timeout: 5s
retries: 5
start_period: 30s
volumes:
- zeebe:/usr/local/zeebe/data
networks:
- camunda-internal
depends_on:
- elasticsearch
- identity

operate: # Docker | Camunda 8 Docs
image: camunda/operate:${CAMUNDA_PLATFORM_VERSION}
container_name: operate
ports:
- “8079:8080”
environment: # Configuration | Camunda 8 Docs
- CAMUNDA_OPERATE_ZEEBE_GATEWAYADDRESS=zeebe:26500
- ZEEBE_CLIENT_ID=${ZEEBE_CLIENT_ID}
- ZEEBE_CLIENT_SECRET=${ZEEBE_CLIENT_SECRET}
- ZEEBE_TOKEN_AUDIENCE=zeebe-api
- ZEEBE_AUTHORIZATION_SERVER_URL=http://keycloak:8080/auth/realms/camunda-platform/protocol/openid-connect/token
- CAMUNDA_OPERATE_ELASTICSEARCH_URL=http://elasticsearch:9200
- CAMUNDA_OPERATE_ZEEBEELASTICSEARCH_URL=http://elasticsearch:9200
# For more information regarding configuration with Identity see:
# Authentication and authorization | Camunda 8 Docs
- SPRING_PROFILES_ACTIVE=identity-auth
- CAMUNDA_OPERATE_IDENTITY_BASEURL=http://${HOST}:8084
- CAMUNDA_OPERATE_IDENTITY_ISSUER_URL=http://${HOST}:18080/auth/realms/camunda-platform
- CAMUNDA_OPERATE_IDENTITY_ISSUER_BACKEND_URL=http://keycloak:8080/auth/realms/camunda-platform
- CAMUNDA_OPERATE_IDENTITY_CLIENTID=operate
- CAMUNDA_OPERATE_IDENTITY_CLIENTSECRET=XALaRPl5qwTEItdwCMiPS62nVpKs7dL7
- CAMUNDA_OPERATE_IDENTITY_AUDIENCE=operate-api
- CAMUNDA_OPERATE_MULTITENANCY_ENABLED=${MULTI_TENANCY_ENABLED}
- SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_ISSUER_URI=http://${HOST}:18080/auth/realms/camunda-platform
- SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_JWK_SET_URI=http://${HOST}:18080/auth/realms/camunda-platform/protocol/openid-connect/certs
- CAMUNDA_OPERATE_IDENTITY_RESOURCEPERMISSIONSENABLED=${RESOURCE_AUTHORIZATIONS_ENABLED}
- management.endpoints.web.exposure.include=health
- management.endpoint.health.probes.enabled=true
- ZEEBE_CLIENT_CONFIG_PATH=/tmp/zeebe_auth_cache
healthcheck:
test: [ “CMD-SHELL”, “wget -O - -q ‘http://localhost:8080/actuator/health/readiness’” ]
interval: 30s
timeout: 1s
retries: 5
start_period: 30s
volumes:
- operate_tmp:/tmp
networks:
- camunda-internal
depends_on:
- zeebe
- identity
- elasticsearch

tasklist: # Docker | Camunda 8 Docs
image: camunda/tasklist:${CAMUNDA_PLATFORM_VERSION}
container_name: tasklist
ports:
- “8082:8080”
environment: # Configuration | Camunda 8 Docs
- CAMUNDA_TASKLIST_ZEEBE_GATEWAYADDRESS=${HOST}:26500
- ZEEBE_CLIENT_ID=${ZEEBE_CLIENT_ID}
- ZEEBE_CLIENT_SECRET=${ZEEBE_CLIENT_SECRET}
- ZEEBE_CLIENT_CONFIG_PATH=/tmp/zeebe_auth_cache
- ZEEBE_TOKEN_AUDIENCE=zeebe-api
- ZEEBE_AUTHORIZATION_SERVER_URL=http://${HOST}:18080/auth/realms/camunda-platform/protocol/openid-connect/token
- CAMUNDA_TASKLIST_ELASTICSEARCH_URL=http://${HOST}:9200
- CAMUNDA_TASKLIST_ZEEBEELASTICSEARCH_URL=http://${HOST}:9200
# For more information regarding configuration with Identity see:
# Authentication | Camunda 8 Docs
- SPRING_PROFILES_ACTIVE=identity-auth
- CAMUNDA_TASKLIST_IDENTITY_BASEURL=http://${HOST}:8084
- CAMUNDA_TASKLIST_IDENTITY_ISSUER_URL=http://${HOST}:18080/auth/realms/camunda-platform
- CAMUNDA_TASKLIST_IDENTITY_ISSUER_BACKEND_URL=http://${HOST}:18080/auth/realms/camunda-platform
- CAMUNDA_TASKLIST_IDENTITY_CLIENTID=tasklist
- CAMUNDA_TASKLIST_IDENTITY_CLIENTSECRET=XALaRPl5qwTEItdwCMiPS62nVpKs7dL7
- CAMUNDA_TASKLIST_IDENTITY_AUDIENCE=tasklist-api
- CAMUNDA_TASKLIST_MULTITENANCY_ENABLED=${MULTI_TENANCY_ENABLED}
- SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_ISSUER_URI=http://${HOST}:18080/auth/realms/camunda-platform
- SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_JWK_SET_URI=http://${HOST}:18080/auth/realms/camunda-platform/protocol/openid-connect/certs
- CAMUNDA_TASKLIST_IDENTITY_RESOURCE_PERMISSIONS_ENABLED=${RESOURCE_AUTHORIZATIONS_ENABLED}
- management.endpoints.web.exposure.include=health
- management.endpoint.health.probes.enabled=true
healthcheck:
test: [ “CMD-SHELL”, “wget -O - -q ‘http://localhost:8080/actuator/health/readiness’” ]
interval: 30s
timeout: 1s
retries: 5
start_period: 30s
volumes:
- tasklist_tmp:/tmp
networks:
- camunda-internal
depends_on:
zeebe:
condition: service_started
elasticsearch:
condition: service_healthy
identity:
condition: service_healthy

connectors: # Overview | Camunda 8 Docs
image: camunda/connectors-bundle:${CAMUNDA_CONNECTORS_VERSION}
container_name: connectors
ports:
- “8085:8080”
environment:
- ZEEBE_CLIENT_BROKER_GATEWAY-ADDRESS=zeebe:26500
- ZEEBE_CLIENT_SECURITY_PLAINTEXT=true
- ZEEBE_CLIENT_ID=${ZEEBE_CLIENT_ID}
- ZEEBE_CLIENT_SECRET=${ZEEBE_CLIENT_SECRET}
- ZEEBE_CLIENT_CONFIG_PATH=/tmp/zeebe_auth_cache
- ZEEBE_TOKEN_AUDIENCE=zeebe-api
- ZEEBE_AUTHORIZATION_SERVER_URL=http://${HOST}:18080/auth/realms/camunda-platform/protocol/openid-connect/token
- CAMUNDA_OPERATE_CLIENT_URL=http://${HOST}:8079
- CAMUNDA_IDENTITY_ISSUER_BACKEND_URL=http://${HOST}:18080/auth/realms/camunda-platform
- CAMUNDA_IDENTITY_CLIENT_ID=connectors
- CAMUNDA_IDENTITY_CLIENT_SECRET=XALaRPl5qwTEItdwCMiPS62nVpKs7dL7
- CAMUNDA_IDENTITY_TYPE=KEYCLOAK
- CAMUNDA_IDENTITY_AUDIENCE=operate-api
- management.endpoints.web.exposure.include=health
- management.endpoint.health.probes.enabled=true
- elasticsearch.disableExporter=false
env_file: connector-secrets.txt
healthcheck:
test: [ “CMD-SHELL”, “curl -f http://localhost:8080/actuator/health/readiness” ]
interval: 30s
timeout: 1s
retries: 5
start_period: 30s
networks:
- camunda-internal
depends_on:
- zeebe
- operate
- identity

optimize: # Docker | Camunda 8 Docs
image: camunda/optimize:${CAMUNDA_OPTIMIZE_VERSION}
container_name: optimize
ports:
- “8083:8090”
environment: # Overview | Camunda 8 Docs
- OPTIMIZE_ELASTICSEARCH_HOST=elasticsearch
- OPTIMIZE_ELASTICSEARCH_HTTP_PORT=9200
- SPRING_PROFILES_ACTIVE=ccsm
- CAMUNDA_OPTIMIZE_ZEEBE_ENABLED=true
- CAMUNDA_OPTIMIZE_ENTERPRISE=false
- CAMUNDA_OPTIMIZE_IDENTITY_ISSUER_URL=http://${HOST}:18080/auth/realms/camunda-platform
- CAMUNDA_OPTIMIZE_IDENTITY_ISSUER_BACKEND_URL=http://${HOST}:18080/auth/realms/camunda-platform
- CAMUNDA_OPTIMIZE_IDENTITY_CLIENTID=optimize
- CAMUNDA_OPTIMIZE_IDENTITY_CLIENTSECRET=XALaRPl5qwTEItdwCMiPS62nVpKs7dL7
- CAMUNDA_OPTIMIZE_IDENTITY_AUDIENCE=optimize-api
- CAMUNDA_OPTIMIZE_IDENTITY_BASE_URL=http://${HOST}:8084
- CAMUNDA_OPTIMIZE_MULTITENANCY_ENABLED=${MULTI_TENANCY_ENABLED}
- CAMUNDA_OPTIMIZE_SECURITY_AUTH_COOKIE_SAME_SITE_ENABLED=false
- CAMUNDA_OPTIMIZE_UI_LOGOUT_HIDDEN=true
- management.endpoints.web.exposure.include=health
- management.endpoint.health.probes.enabled=true
healthcheck:
test: [ “CMD-SHELL”, “curl -f http://localhost:8090/api/readyz” ]
interval: 30s
timeout: 1s
retries: 5
start_period: 30s
volumes:
- “./.optimize/environment-config.yaml:/optimize/config/environment-config.yaml”
restart: on-failure
networks:
- camunda-internal
depends_on:
- identity
- elasticsearch

identity: # Docker | Camunda 8 Docs
container_name: identity
image: camunda/identity:${CAMUNDA_PLATFORM_VERSION}
ports:
- “8084:8084”
environment: # Configuration variables | Camunda 8 Docs
SERVER_PORT: 8084
IDENTITY_RETRY_DELAY_SECONDS: 30
KEYCLOAK_URL: http://${HOST}:18080/auth
IDENTITY_AUTH_PROVIDER_ISSUER_URL: http://${HOST}:18080/auth/realms/camunda-platform
IDENTITY_URL: http://${HOST}:8084
IDENTITY_BASE_PATH: /
IDENTITY_AUTH_PROVIDER_BACKEND_URL: http://${HOST}:18080/auth/realms/camunda-platform
IDENTITY_DATABASE_HOST: postgres
IDENTITY_DATABASE_PORT: 5432
IDENTITY_DATABASE_NAME: bitnami_keycloak
IDENTITY_DATABASE_USERNAME: bn_keycloak
IDENTITY_DATABASE_PASSWORD: “#3]O?4RGj)DE7Z!9SA5”
KEYCLOAK_INIT_OPERATE_SECRET: XALaRPl5qwTEItdwCMiPS62nVpKs7dL7
KEYCLOAK_INIT_OPERATE_ROOT_URL: http://${HOST}:8079
KEYCLOAK_INIT_TASKLIST_SECRET: XALaRPl5qwTEItdwCMiPS62nVpKs7dL7
KEYCLOAK_INIT_TASKLIST_ROOT_URL: http://${HOST}:8082
KEYCLOAK_INIT_OPTIMIZE_SECRET: XALaRPl5qwTEItdwCMiPS62nVpKs7dL7
KEYCLOAK_INIT_OPTIMIZE_ROOT_URL: http://${HOST}:8083
KEYCLOAK_INIT_WEBMODELER_ROOT_URL: http://${HOST}:8070
KEYCLOAK_INIT_CONNECTORS_SECRET: XALaRPl5qwTEItdwCMiPS62nVpKs7dL7
KEYCLOAK_INIT_CONNECTORS_ROOT_URL: http://${HOST}:8085
KEYCLOAK_INIT_ZEEBE_NAME: zeebe
KEYCLOAK_USERS_0_USERNAME: “demo”
KEYCLOAK_USERS_0_PASSWORD: “demo”
KEYCLOAK_USERS_0_FIRST_NAME: “demo”
KEYCLOAK_USERS_0_EMAIL: “demo@acme.com”
KEYCLOAK_USERS_0_ROLES_0: “Identity”
KEYCLOAK_USERS_0_ROLES_1: “Optimize”
KEYCLOAK_USERS_0_ROLES_2: “Operate”
KEYCLOAK_USERS_0_ROLES_3: “Tasklist”
KEYCLOAK_USERS_0_ROLES_4: “Web Modeler”
KEYCLOAK_CLIENTS_0_NAME: zeebe
KEYCLOAK_CLIENTS_0_ID: ${ZEEBE_CLIENT_ID}
KEYCLOAK_CLIENTS_0_SECRET: ${ZEEBE_CLIENT_SECRET}
KEYCLOAK_CLIENTS_0_TYPE: M2M
KEYCLOAK_CLIENTS_0_PERMISSIONS_0_RESOURCE_SERVER_ID: zeebe-api
KEYCLOAK_CLIENTS_0_PERMISSIONS_0_DEFINITION: write:*
MULTITENANCY_ENABLED: ${MULTI_TENANCY_ENABLED}
RESOURCE_PERMISSIONS_ENABLED: ${RESOURCE_AUTHORIZATIONS_ENABLED}
healthcheck:
test: [ “CMD”, “wget”, “-q”, “–tries=1”, “–spider”, “http://localhost:8082/actuator/health” ]
interval: 5s
timeout: 15s
retries: 30
start_period: 60s
restart: on-failure
volumes:
- keycloak-theme:/app/keycloak-theme
networks:
- camunda-internal
depends_on:
keycloak:
condition: service_healthy

postgres: # Docker
container_name: postgres
image: postgres:${POSTGRES_VERSION}
environment:
POSTGRES_DB: bitnami_keycloak
POSTGRES_USER: bn_keycloak
POSTGRES_PASSWORD: “#3]O?4RGj)DE7Z!9SA5”
restart: on-failure
healthcheck:
test: [ “CMD-SHELL”, “pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}” ]
interval: 10s
timeout: 5s
retries: 5
networks:
- camunda-internal
volumes:
- postgres:/var/lib/postgresql/data

keycloak: # Docker
container_name: keycloak
image: bitnami/keycloak:${KEYCLOAK_SERVER_VERSION}
volumes:
- keycloak-theme:/opt/bitnami/keycloak/themes/identity
ports:
- “18080:8080”
environment:
KEYCLOAK_HTTP_RELATIVE_PATH: /auth
KEYCLOAK_DATABASE_HOST: postgres
KEYCLOAK_DATABASE_PASSWORD: “#3]O?4RGj)DE7Z!9SA5”
KEYCLOAK_ADMIN_USER: admin
KEYCLOAK_ADMIN_PASSWORD: admin
restart: on-failure
healthcheck:
test: [ “CMD”, “curl”, “-f”, “http://localhost:8080/auth” ]
interval: 30s
timeout: 15s
retries: 5
start_period: 30s
networks:
- camunda-internal
depends_on:
- postgres

elasticsearch: # Docker
image: docker.elastic.co/elasticsearch/elasticsearch:${ELASTIC_VERSION}
container_name: elasticsearch
ports:
- “9200:9200”
- “9300:9300”
environment:
- bootstrap.memory_lock=true
- discovery.type=single-node
- xpack.security.enabled=false
# allow running with low disk space
- cluster.routing.allocation.disk.threshold_enabled=false
- elasticsearch.disableExporter=false
- “ES_JAVA_OPTS=-Xms512m -Xmx512m”
ulimits:
memlock:
soft: -1
hard: -1
restart: always
healthcheck:
test: [ “CMD-SHELL”, “curl -f http://localhost:9200/_cat/health | grep -q green” ]
interval: 30s
timeout: 5s
retries: 3
networks:
- camunda-internal
volumes:
- elastic:/usr/share/elasticsearch/data

kibana:
image: docker.elastic.co/kibana/kibana:${ELASTIC_VERSION}
container_name: kibana
ports:
- 5601:5601
volumes:
- kibana:/usr/share/kibana/data
depends_on:
- elasticsearch
networks:
- camunda-internal
profiles:
- kibana

volumes:
zeebe:
elastic:
postgres:
keycloak-theme:
kibana:
operate_tmp:
tasklist_tmp:

networks:
camunda-internal:

Praveen_Kumar_Reddy · April 12, 2024, 5:08am

Hi @pacman8,

In the operate the data is not getting populated

Please check whether elasticsearch indices are created and getting updated on deploying, starting instances.

The communication between ES and optimize is established that is the reason you are able see the data
similarly make sure your operate is connected with elasticsearch

In the configuration use container name instead of host for URL and BASEURL type configuration
eg: CAMUNDA_OPERATE_ELASTICSEARCH_URL: http://elasticsearch:9200

CAMUNDA_OPERATE_IDENTITY_BASEURL: http://identity:8084

Hope this might be useful

pacman8 · April 12, 2024, 12:02pm

Hi @Praveen_Kumar_Reddy

Unfortunately not. Operate creates indices ("operate-*) in eleasticsearch, so a connection is established. Data is filled in too. As soon as I start my job worker my process gets deployed and is listed in “operate-process-8.3.0_”. A new process instance entry is shown in “operate-metric-8.3.0_” as soon as I call my webhook.

Praveen_Kumar_Reddy · April 17, 2024, 4:35am

Hi @pacman8 ,
I’m out of ideas and it seems a bit strange for me,
Need to recreate the issue and debug it and unfortunately server is unavailabe for me to test right now

pacman8 · April 17, 2024, 1:07pm

Hi @Praveen_Kumar_Reddy
Thank you for keeping up investigation in that case.
In between I found the problem, maybe. In short there is a high possibility, that my docker remote host internal networking system is screwed up.
Today I received a laptop from my infrastructure department on which I installed an ubuntu, docker and camunda 8.5.0. I used an unmodified docker-compose.yaml and set HOST variable in .env file → operate worked. I installed camunda 8.5.0 on my old remote host with same docker-compose.yaml and .env file settings → operate is still not working. Its an old docker host with a lot other containers runnning on that.
Anyway, the camunda part seems to be solved.