Process variables and task availability issue

Camunda 8 Topics
1

Hi there,
I’m using Self-Managed Camunda, version 8.6.11 deployed using Helm Charts and I have one instance (for now) of ElasticSearch database.

We are considering switching to commercial version soon but before that we’ve encountered a few major UI issues because we have a process that have “a lot” of user tasks. Variables from one task has to be used in the second task and so on but we have to wait for some time for them to because available. We are using Zeebe User Tasks, but we also use Tasklist API to query the tasks because Zeebe API for fetching the tasks is still experimental in this version.

We’ve tried to reduce the sync time between ElasticSearch and other components (Zeebe, Operate and Tasklist) but it didn’t help much as for full synchronization we need to wait 5-8 seconds which is not acceptable for end users. We’ve set the env variable ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_BULK_DELAY to 1 second but it didn’t help much.

Do you know any kind of approach on how this can be solved? I’ve used Camunda 7 before and this wasn’t an issue. I understand that Camunda 8 is different from aspect of architecture but still this is the basic use case.

Thank you in advance!

2

There are few other parameters need to be tuned to increase the exporter frequency to reduce the latency.

bulk:
delay: 5
size: 1000
memoryLimit: 10485760

Please tune all the parameters and try out.

3

Delay is already 1 (minimum possible value), so I don’t think that other params are relevant, right?

4

Other parameters are also relevant. I would recommend to modify the value as per your requirement and try it out.

5

Could you please elaborate more on how other parameters are relevant if we’ve already configured delay to 1 second?

In my opinion, the flush will happen if any of the three conditions are met:

  • the delay elapses
  • the batch size is reached or
  • the memory limit is reached

If you have any additional info I would like to hear.
Thanks.

6

I shared the document, if you look at the document, it will aggregate all 3 values. Please take a look at the document link.

With the default configuration, the exporter will aggregate records and flush them to Elasticsearch:

  1. When it has aggregated 1000 records.
  2. When the batch memory size exceeds 10 MB.
  3. Five seconds have elapsed since the last flush (regardless of how many records were aggregated).
7

Is there any alternative solution for this case or just this one?

8

Hi @heril.muratovic06 - because of Zeebe’s asynchronous and eventually consistent event driven design, there will always be at least a slight delay between an action occurring in the process and the data being exported to Operate and the APIs. However, there are some improvements to this coming in the near future:

  • we are working on user task listeners (search “user task listener” on roadmap.camunda.com to see all the features); for most use cases needing near-real-time data about user tasks, user task listeners will likely be a good solution. These are targeted for release in 8.8 later this year.
  • we are also working on a real time, read-only API within Zeebe to fetch process data directly, bypassing the exporter for some process data.

From your description, I assume you’re using a custom tasklist application and that’s why the export delay is causing headaches in your frontend. Are you using Camunda Forms as well, or are your forms built into your application? While it isn’t the best idea, and not semantic BPMN, you could also look at replacing your user tasks with service tasks and pushing the data to your application via job workers.

Another user I worked with had similar issues when building a custom frontend, and this thread has a lot of good information as well: There's a delay after a process instance is started. They went with the job worker route until user task listeners are generally available.

1 Like
9

Hi @nathan.loding, yes we’re using custom tasklist application but we’re using Camunda Forms as well. Our tasks are Zeebe User Tasks.

So, your recommendation will be the following (correct me if I’m wrong):

  1. Replace Zeebe User Tasks with Service Tasks (job workers)
  2. Somehow we need to provide a formId to this service task (maybe some input variable or something similar)
  3. Create a worker that will read this formId and get the schema from Tasklist API or just JSON from the system (*.form file)
  4. Notify Frontend App via websocket to render the form and move on (this can be tricky because what will happen in case of refreshing the page etc.); we need to come up with some better solution

Should we put Zeebe User Task after job worker fetches the form so we can complete the task and stop the process until the task is completed?

I’m trying to imagine the situation that’s why I’m asking :slight_smile:

10

@heril.muratovic06 - if you want to switch to service tasks, I would embed the form into your application rather than fetching it from Camunda, then the job worker gets the variables for your application. Camunda’s forms are supported by the open source form-js library so they are easy to render in your app. There’s some obvious downsides to that approach, so it’s probably not the best workaround for your particular situation. (One downside being that editing a form in Modeler requires several more steps to get that form into your application.)

The best solution will be available in 8.8 with the user task listeners, but until then, it’s choosing a temporary workaround.

11

Hi @nathan.loding, thank you for the answer.

Btw, is it possible to implement “wait state” in Camunda 8 but not using User Tasks? And then to trigger that state on demand, so the process can move forward?

12

@heril.muratovic06 - probably the best way to approach that is message events. You would add an intermediate message catch event to your process, and the process will wait there until it receives a message with a given key. Once that message is received, it will continue. That message can come from another process or via the API.

1 Like
13

At the end, we’ve decided to replace all User Tasks with Intermediate Message Catch Events and to add “Start” execution listener on each message event. Listener is responsible for creating a user task for specific user/group but it will persist the task in the local database. By this, we’re completely avoiding a usage of the Tasklist API. The task completion logic is replaced by message correlation.

Additionally, all variables are stored directly in the local database. We’ve configured AOP for all workers that are persisting some variables to Zeebe so we can intercept them and store/update them in the local database. By this, we’re completely avoiding usage of the Operate API.

We’ve managed to test this solution and I can confirm that it’s working and there are no delays.

If someone needs to hear more about the implementation details, feel free to contact me.

1 Like