Camunda 8 Self Managed JobWorker polling stays stucked

Camunda 8 Topics
,
1

Hi everybody,

I was experimenting little bit with C8 and Self Managed.

I installed the C8 in my local Kubernetes with the following Helm Configuration.

camunda-platform:
  global:
    identity:
      auth:
        enabled: false

  identity:
    enabled: false

  optimize:
    enabled: false

  zeebe:
    clusterSize: 1
    partitionCount: 1
    replicationFactor: 1
    pvcSize: 10Gi

  zeebe-gateway:
    replicas: 1

  connectors:
    enabled: true
    inbound:
      mode: disabled

  elasticsearch:
    master:
      replicaCount: 1
      persistence:
        size: 15Gi

And created a Test BPMN.

I defined a BPMN with a simple service task ‘additionalBilling’.

Screenshot 2024-03-21 at 11.19.53
Screenshot 2024-03-21 at 11.19.531944×870 80.1 KB

Screenshot 2024-03-21 at 11.23.15
Screenshot 2024-03-21 at 11.23.151242×272 42.9 KB

And I have a JobWorker for this Service Task in a Spring Boot Application.

@Component
@Slf4j
public class ExtraBillingFacade {
    @JobWorker(type = "additionalBilling", pollInterval = 1000L)
    public void doSomething(final ActivatedJob job) {
        log.info("We are doing something! [{}]", job);
    }
}

which has a following ‘application.yml’ configuration.

zeebe:
  client:
    security:
      plaintext: "true"
    broker:
      gateway-address: "camunda-platform-zeebe-gateway.camunda:26500"
    connection-mode: "ADDRESS"
    request-timeout: 10s

camunda:
  operate:
    client:
      base-url: "camunda-platform-operate.camunda:8080"
      enabled: true
      url: "camunda-platform-operate.camunda:8080"
      username: "demo"
      password: "demo"

tasklist:
  client:
    base-url: "camunda-platform-tasklist.camunda:8080"
    url: "camunda-platform-tasklist.camunda:8080"
    username: "demo"
    password: "demo"

And I see in the Spring Boot Logs and Operate the BPMN is deployed, Workers configured/started and workers started polling (my workers are also in the k8s Cluster as Pods).

Configuring 1 Zeebe worker(s) of bean 'scoreFacade': [ZeebeWorkerValue{type='ProcessFraudPreventionRequest', name='scoreFacade#processFraudPreventionMessage', timeout=-1, maxJobsActive=-1, requestTimeout=-1, pollInterval=-1, autoComplete=true, fetchVariables=[], enabled=true, methodInfo=io.camunda.zeebe.spring.client.bean.MethodInfo@241fc278}]
2024-03-21T10:06:11.810Z  INFO 1 --- [workflow] [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 13 endpoint(s) beneath base path '/actuator'
2024-03-21T10:06:13.096Z DEBUG 1 --- [workflow] [           main] c.p.OperateClientConfigurationProperties : Connecting to Camunda Operate on URL: camunda-platform-operate.camunda:8080
2024-03-21T10:06:15.014Z  INFO 1 --- [workflow] [           main] o.s.b.web.embedded.netty.NettyWebServer  : Netty started on port 8080
2024-03-21T10:06:18.697Z  INFO 1 --- [workflow] [           main] c.a.p.ZeebeDeploymentAnnotationProcessor : Deployed: <Definitions_01hqw9d:1>,<fraud-prevention:1>,<customer_order:1>
2024-03-21T10:06:18.735Z  INFO 1 --- [workflow] [           main] i.c.z.s.c.jobhandling.JobWorkerManager   : . Starting Zeebe worker: ZeebeWorkerValue{type='addCustomerToBilling', name='billingFacade#addCustomerToBilling', timeout=-1, maxJobsActive=-1, requestTimeout=-1, pollInterval=-1, autoComplete=true, fetchVariables=[], enabled=true, methodInfo=io.camunda.zeebe.spring.client.bean.MethodInfo@4b98225c}
2024-03-21T10:06:18.737Z  INFO 1 --- [workflow] [           main] i.c.z.s.c.jobhandling.JobWorkerManager   : . Starting Zeebe worker: ZeebeWorkerValue{type='addCustomerToCRM', name='CRMFacade#addCustomerToCrm', timeout=-1, maxJobsActive=-1, requestTimeout=-1, pollInterval=-1, autoComplete=true, fetchVariables=[], enabled=true, methodInfo=io.camunda.zeebe.spring.client.bean.MethodInfo@4beae1e3}
2024-03-21T10:06:18.738Z  INFO 1 --- [workflow] [           main] i.c.z.s.c.jobhandling.JobWorkerManager   : . Starting Zeebe worker: ZeebeWorkerValue{type='additionalBilling', name='extraBillingFacade#doSomething', timeout=-1, maxJobsActive=-1, requestTimeout=-1, pollInterval=1000, autoComplete=true, fetchVariables=[], enabled=true, methodInfo=io.camunda.zeebe.spring.client.bean.MethodInfo@3c488b34}
2024-03-21T10:06:18.739Z  INFO 1 --- [workflow] [           main] i.c.z.s.c.jobhandling.JobWorkerManager   : . Starting Zeebe worker: ZeebeWorkerValue{type='provisionSIM', name='SIMCardProvisionerFacade#provisionSIMCard', timeout=-1, maxJobsActive=-1, requestTimeout=-1, pollInterval=-1, autoComplete=true, fetchVariables=[], enabled=true, methodInfo=io.camunda.zeebe.spring.client.bean.MethodInfo@32e697ac}
2024-03-21T10:06:18.740Z  INFO 1 --- [workflow] [           main] i.c.z.s.c.jobhandling.JobWorkerManager   : . Starting Zeebe worker: ZeebeWorkerValue{type='registerSIM', name='SIMCardRegistryFacade#registerSimCard', timeout=-1, maxJobsActive=-1, requestTimeout=-1, pollInterval=-1, autoComplete=true, fetchVariables=[], enabled=true, methodInfo=io.camunda.zeebe.spring.client.bean.MethodInfo@3866c96e}
2024-03-21T10:06:18.740Z  INFO 1 --- [workflow] [           main] i.c.z.s.c.jobhandling.JobWorkerManager   : . Starting Zeebe worker: ZeebeWorkerValue{type='ProcessFraudPreventionRequest', name='scoreFacade#processFraudPreventionMessage', timeout=-1, maxJobsActive=-1, requestTimeout=-1, pollInterval=-1, autoComplete=true, fetchVariables=[], enabled=true, methodInfo=io.camunda.zeebe.spring.client.bean.MethodInfo@241fc278}
2024-03-21T10:06:18.899Z TRACE 1 --- [workflow] [pool-2-thread-1] io.camunda.zeebe.client.job.poller       : Polling at max 32 jobs for worker billingFacade#addCustomerToBilling and job type addCustomerToBilling
2024-03-21T10:06:18.941Z TRACE 1 --- [workflow] [pool-2-thread-1] io.camunda.zeebe.client.job.poller       : Polling at max 32 jobs for worker CRMFacade#addCustomerToCrm and job type addCustomerToCRM
2024-03-21T10:06:18.942Z TRACE 1 --- [workflow] [pool-2-thread-1] io.camunda.zeebe.client.job.poller       : Polling at max 32 jobs for worker SIMCardProvisionerFacade#provisionSIMCard and job type provisionSIM
2024-03-21T10:06:18.943Z TRACE 1 --- [workflow] [pool-2-thread-1] io.camunda.zeebe.client.job.poller       : Polling at max 32 jobs for worker SIMCardRegistryFacade#registerSimCard and job type registerSIM
2024-03-21T10:06:18.943Z TRACE 1 --- [workflow] [pool-2-thread-1] io.camunda.zeebe.client.job.poller       : Polling at max 32 jobs for worker scoreFacade#processFraudPreventionMessage and job type ProcessFraudPreventionRequest
2024-03-21T10:06:19.004Z  INFO 1 --- [workflow] [           main] o.s.c.workflow.ThrowoffApplication       : Started ThrowoffApplication in 32.446 seconds (process running for 35.913)
2024-03-21T10:06:19.739Z TRACE 1 --- [workflow] [pool-2-thread-1] io.camunda.zeebe.client.job.poller       : Polling at max 32 jobs for worker extraBillingFacade#doSomething and job type additionalBilling

then I started the process and I see in the logs and operate in the Process Instance created.

INFO 1 --- [workflow] [or-http-epoll-4] o.s.c.w.adapter.RestControllerAdapter    : Staring process
2024-03-21T10:09:25.847Z  INFO 1 --- [workflow] [or-http-epoll-4] o.s.c.w.core.CustomerOnboardingFacade    : Triggering process
2024-03-21T10:09:25.939Z  INFO 1 --- [workflow] [or-http-epoll-4] o.s.c.w.core.CustomerOnboardingFacade    : Started instance for processDefinitionKey=2251799813685858, bpmnProcessId=customer_order, version=1 with processInstanceKey=2251799813686166

but it is never picked with the poll and or poll even does not return with Timeout.

Screenshot 2024-03-21 at 11.39.46
Screenshot 2024-03-21 at 11.39.46938×754 44.2 KB

Screenshot 2024-03-21 at 11.40.04
Screenshot 2024-03-21 at 11.40.04916×582 68.6 KB

I have no idea what is the reason why Task is not picked by JobWorker or why the pollers are stucked, even that I configured a timeout.

If I understand the documentation of JobWorkers
Job Workers
correctly, Jobs are pulled on jobstreams from zeebe.

When I call the actuator from zebee
GET http://camunda-platform:51468/actuator/jobstreams

Even that I see the process instance in ‘operate’, jobstream on zeebe delivers empty lists

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 25

{
  "remote": [],
  "client": []
}

Any hints?

3

Ok, if anybody stucks with this problem, after little bit fight I discovered the problem…

Naturally I was using

io.camunda:spring-zeebe-starter

as Camunda Starter for Spring Boot

and I thought it would be good idea to use

org.springframework.boot:spring-boot-starter-pulsar

also for Apache Pulsar functionality.

This two dependencies does not like each other, I could still not figure out the underlying condition, but the JobWorker Polling stucks if both our under the classpath at the same time.

If I use Apache Pulsar native client API everything works perfectly.

If anybody needs it, you can see my solution here.

4

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.