Hi @Rania_Chakroun !
On a high level, exporters are explained here: (Exporters | Camunda 8 Docs).
To get started, it’s best to look at the projects from the community: Repositories · GitHub
The exporter config allow you to define the batch size and time yourself. You can, for example, define an export time of 1 second, which means events are exported every second.
Note that the performance of Zeebe can decrease when export processing is too compute-intense. That’s why the default is 5 seconds as we give guarantees on processing speed before on output speed. But here we are speaking about cutting-edge processing like financial transaction use cases where it goes into the millions per second.
Best,
Christian
2 Likes
@MagzhanUnited - C7 is still a synchronous engine, yes; however, it is an end-of-life product, with development ending next year with only enterprise accounts receiving security updates after.
I was a bit vague with what new features are coming in 8.6+ - I’m going to talk with our product team and see what can be shared in advance. I don’t want to accidentally make any false promises.
Ok, everyone, thanks for your replies. Now I understand the nature of Camunda 8.
We still would be happy if there’s a solution to the problem, and yesterday I was informed that Zeebee 8.5 introduces Zeebee User Tasks. With Zeebe User Tasks we can directly query the workflow engine for the status of the task. However, searching for tasks is still done with the Tasklist API, so it doesn’t solve our problem.
Hi @kristoffer.jalen - I saw your email chain also, but thought a more public reply here would be beneficial. You are correct, searching for tasks is still enabled by Tasklist, which requires the data to be exported before the task will appear in the search results. As of right now, that architecture isn’t changing in the 8.6 release. However, you might notice a spoiler in the documentation (here and here):
Task listeners are scheduled to be released in 8.6. (This is one of the “some changes are coming in 8.6 but I’m not sure what I can publicly comment on yet” features I was alluding to earlier! They will support Zeebe User Tasks, but I admit I am not sure the full scope of the feature and what other task types may or may not be supported.) Task listeners will help a lot with what you’re trying to achieve, and do it in an event-driven manner, rather than needing your application to query the process.
In addition to what Nathan has written, there will also be real-time GET endpoints in 8.6, allowing to retrieve a task after creation by it’s ID.
I solved this problem with job worker that directly inserts data into elasticsearch without Zeebe
2 Likes
@nathan.loding, in There's a delay after a process instance is started - #9 by nathan.loding, you say:
I would recommend not using a User Task, and instead use a Service Task with either a job worker or Connector to push the task to your application, rather than your application trying to pull it.
I don’t understand how to model, nor how to implement, this. You mentioned using a Service Task. I can’t see how that can work since I need a form. A form must be associated to a process definition via a User Task. I tried using a Service Task and added a “customer header” with the formKey as value, but that obviously doesn’t work, because the form cannot be found unless it’s associated with a User Task.
If I would use a user task implemented as a job worker, and then my worker is actually able to extract the processDefinitionKey, the taskId, and the formKey. These three values are those I need to get the form, and to complete the task. The worker can send these values to the browser using WebSockets. Is that a good idea?
Can I use a Zeebe user task? But then how should my model push the data to the application? There is no worker for a Zeebe user task. Should I use Zeebe user tasks in combination with a Service Task that sends an event to my application via some event streaming platform, like Kafka or Azure Event Hub? Is that what you were suggesting here:
A similar “push” method can be used for events beyond user tasks; for instance, if your application needs to know when a process has ended, you can include a final service task right before the end event that notifies your application.
@kristoffer.jalen - good questions! Of course the answer is “it depends,” but let me try to answer a few of your questions:
- You are correct, when using a service task you can’t easily pass a form via ID. Because the forms are just JSON templates, and you are using a custom task list UI (instead of using Tasklist), you do not necessarily have to use user tasks. My suggestion was meant as an alternative approach; it may not be strictly semantic BPMN but may work for your use case and requirements.
- You can implement a job worker that listens for user tasks (if you search the forum, there are several people who have done this), but the downside is that your job worker is triggered for every user task. This means you need to implement logic in a single worker to handle all the possible user tasks in your process. It’s not my favorite approach but it is workable.
- Zeebe user tasks do not work with job workers. But a Zeebe user task has its lifecycle managed by Zeebe, rather than managed by Tasklist, which has some advantages; however, the feature you are really looking for is planned for the 8.6 release in October, which is the real-time GET endpoints mentioned by Christian.
- Regarding the “push method” I mentioned, you don’t necessarily need Kafka or Event Hub. It was meant to show that you don’t need to query the process all the time, and instead you can use other tasks (such a REST Connector, or Kafka Connector, or just a service task with a job worker) to send data to your application. This bypasses the exporter and any delays in data availability outside Zeebe and sends the information directly to your application as soon as the process gets to that task.
Hope that helps, but please let me know what other questions you have! And hopefully we’ll have a chance to connect at CamundaCon!
1 Like
@nathan.loding Thank you for your prompt response. This topic is of significant interest to me, and I am committed to understanding it thoroughly. I appreciate your patience as I continue to delve into this matter.
Just to be clear, what I want:
- Start the process (either by a webhook or Zeebeclient; it doesn’t matter).
- Instantly inform the user (browser) about the next task and its form key.
- The browser fetches the form and renders it.
- The user submits the form, completing the task.
- The token moves to the next task, repeating steps 2-5.
- After n forms, the whole process is completed.
So:
-
I can’t use a User Task. This is due to Camunda 8’s event-driven nature, which would necessitate the user querying (polling) the Tasklist API for up to 7 seconds, and sometimes even longer.
-
I can’t use a Zeebe User Task because I can’t GET the task and cannot handle it with a worker.
-
I can’t use a Service Task because a Service Task cannot be associated with a form, and a deployed form cannot be found (Get Form API) without being associated with a User Task.
-
If I manage to let a User Task publish a message (if that’s possible) saying, ‘Hey, the token is here, and this is the task ID and form ID,’ then my application can catch that message and transmit that data to the client over a WebSocket connection. However, this approach also has limitations as I need to wait seven seconds before the user can complete the task, which is unacceptable for the user experience.
You say:
you can use other tasks (such a REST Connector, or Kafka Connector, or just a service task with a job worker) to send data to your application. This bypasses the exporter and any delays in data availability outside Zeebe and sends the information directly to your application as soon as the process gets to that task.
But that I don’t understand. Please show me how I can model that. Where should such a Service Task be placed? Which activity should hold and provide the form definition?
This use case is straightforward. I am exploring the feasibility of orchestrating hundreds of business processes. While there may be more optimal solutions than Camunda, Camunda is so strong in the orchestration part that I prefer Camunda. What baffles me a bit is that, on the one hand, I get a fantastic impression of Camunda, both in terms of all of you giving this professional support, all the community participation, and all the resources on documentation, blog posts, the CamundaCon conference, and so on. On the other hand, I scratch my head to understand the potential business opportunities Camunda loses if this simple use case isn’t supported. It would be supported without hassle if the Start event immediately returned the next task without the exporting and indexing in ES. I may have misunderstood something. 
Don’t get me wrong, I’m not debating; I’m asking. I’m eager to learn and understand because I find Camunda attractive.
What should I do? Are there any sample projects I can look at?
And yes, I very much look forward to CamundaCon in two weeks!
@kristoffer.jalen - does your process do anything other than move between user tasks?
@nathan.loding - Yes, the process also has a DMN table, conditional moves, error handling and service tasks.
@kristoffer.jalen - are the DMN and service tasks mixed in with the user tasks? Or are the user tasks together and when the user has finished filling in the data, additional tasks are then executed?
The user tasks are together and when the user has finished filling in the data, additional tasks are then executed.
And also: it has been suggested you can use a custom exporter, and bypass the exporter and any delays in data availability outside Zeebe. But custom exporters are not available in SaaS, hence self-managed would be the only option running Camunda.
@kristoffer.jalen - correct, custom exporters are only available in Self Managed right now.
If your user tasks are all together with no additional activities between them, then you are probably better served handling the forms entirely inside your application then starting the process. For instance, here’s a basic example process I just drew:
Forms 1, 2, 3a, and 3b would be best served from your application and removed from the process. There are some assumptions of course: these are all forms to be completed by a single user and real-time switching between forms is a requirement (similar to a multi-step form wizard). If the delay before it loads in Tasklist is acceptable, or if they are being handled by different departments/users (which negates any delays in Tasklist), then using them as part of the process makes sense.
There are opportunities to make Camunda act as a router, but process orchestration is a much larger concept than that. More real time APIs are coming 8.6, as well some larger architectural changes that will help lay a foundation for building out new features in the future.
