The "Decoupled Job Completion" pattern in C# client

Camunda 8 Topics Discussion & Questions
1

I am referring to https://developer.aliyun.com/mirror/npm/package/zeebe-node-next/v/2.4.0 - scroll down to the " Decoupled Job Completion pattern" section. It’s a good overview of what I am trying to implement using a C# client. Another good reference is https://blog.bernd-ruecker.com/orchestrating-azure-functions-using-bpmn-and-camunda-a-case-study-ff71264cfad6 where Bernd mentions separate Fetcher and Completer connectors (albeit for Camunda but the concept is exactly what I am looking to do with RabbitMQ or Async Azure function calls - and yes, I’ll maintain the job key for correlating it back). The Node client above has this nifty feature:

// Call forwarded() to release worker capacity
complete.forwarded()

I am interested in releasing worker capacity after triggering async operation (e.g. if MaxJobsActive is set to 1, I want this worker to fetch and process the next task). The jobs could be long-running (I have reviewed a long-running jobs topic on this forum already, seems that with a small-ish number of jobs memory consumption won’t be an issue). Back-pressure would be handled by RabbitMQ.

As it is right now, I can probably set the worker’s .Timeout(TimeSpan.FromDays(10)) - and hang out for 10 days waiting for completion. This, however, will count this long-running job against my MaxJobsActive count. And, depending on the size of workflow variable objects, (correct me if I am wrong - I haven’t reviewed C# client code) this would likely eat up the memory on the machine running the client, basically maintaining all the job details including variables in-memory. I would like to free up the resources on the machine running the client as soon as the job is dispatched externally (e.g. via RabbitMQ). There will be a separate process listening for task completion and signalling job completion to the broker.

What are the recommendations to implement the above in C# client and possibly not hog up the worker jobs allotment while awaiting job completion? Did I miss anything in C# client docs, or is this feature not available? I am using the latest version of everything (Zeebe 0.23.1)

1 Like
2

Hi @mostwired,

The answer to your question is: pull request.

You can implement support for the decoupled completion pattern in the C# worker code, and become famous and praised all over the world.

A few people have asked for it. I implemented it in the Node.js client. I know that @Zelldon, the author of the C# client, is happy to take PRs.

Josh

3

Hi @mostwired

Why do you assume that it is not possible? Do have any problems with that? Do you even tried it?

The clients work mostly in the same way, also in this case. The job is not kept in mem or elsewhere. If the handler is done and haven’t completed it it is gone. If you stored the key, you can use the client to complete it later.

If you not use a long activation timeout this means another worker will get this sooner.

Hope that helps.

Greets
Chris

4

@mostwired I got it working in Java and I also have a TODO to implement the forwarded() method for the client to reduce the MaxJobsActive

5

Hi Chris,

I’ve briefly looked at the code in the JobWorker.cs, looks like the job is not flagged complete when handler finishes and doesn’t report job as complete to the broker, unless AutoCompletion on worker is set. The job would get dequeued from workItems (which is good as it stores job’s variables as well). No message will be sent to the broker notifying of job completion. Line 194 of JobWorker.cs makes use of workItems count - which will be reduced on handler task completion regardless of whether the handler explicitly reported job as done:

if (workItems.Count < maxJobsActive)

Is the above correct?

6

Correct. It means it will poll new jobs also and you can complete the job with help of the corresponding key.

1 Like