There's a delay after a process instance is started

Camunda 8 Topics
1

I’m evaluating Camunda 8 with a simple use case:

  • A BPMN model with two User Tasks.
  • A custom built (by me) React UI.
  • This UI should use the Camunda REST API to get the first task, and its form, and then the next task, and its form, with the user completing each task after filling out the form.

But I have an issue with delays:

  1. The user opens the UI, and an HTTP request to an HTTP webhook connector starts a BPMN process instance. The webhook returns the process instance key in the correlation object.
  2. I then pass the process instance key to the Search tasks API to get the first task.
  3. The first task gives me a formId, and I pass this formId to the Get a form API to get the first form.

But, I have to wait up to like ten seconds for step 2, or else the Search tasks API will return an empty result.

I have the same problem in the next step:

  1. When the user has filled out the form, I send the form variables to Complete a task API.
  2. I then pass the process instance key to the Search tasks API to get the second task.
  3. The second task gives me a formId, and I pass this formId to the Get a form API to get the second form.

Am I doing this wrong? Is this not how it should be done? A user cannot wait up to 10 seconds for the forms to load, right?

2

Anyone? It’s a bit of a showstopper for us.

3

Hi @kristoffer.jalen - some delay is expected, yes, though 10 seconds sounds a bit long.

The reason behind the delay is the event stream exporter. Zeebe (the workflow engine) sends an event stream to an exporter, which then writes the events to a data store. By default it uses Elasticsearch. When you query the API, you are actually querying that data store. There are several reasons for this architecture, but the primary one is maintaining Zeebe’s distributed nature and allowing each broker to process tasks “independently”.

If you are using a Self-Managed instance, you can install your own exporter, which could be used to trigger events in your application (“push” rather than “pull”).

However, like I mentioned, 10 seconds is longer than I would expect the delay to regularly take. Are you using SaaS or a Self-Managed instance? It seems like that delay can be greatly minimized, but the answer to “how?” depends on how you’re using Camunda.

4

Very confusing. Should I consider Camunda not appropriate for a customer-facing tasks then? It’s such a simple and common use case that I don’t really understand what I’m missing here:

  1. Create a User task in Camunda
  2. UI: start a process and show the form
  3. UI: let the user submit the form

You are saying that this form is not supposed to be immediately available to the user? Then it can’t be used by others than backoffice users. People expect maybe hundreds of milliseconds, not 7 seconds.

I have tried it multiple times, and I would say it takes approximately 7 seconds until the task is available. I’m using SaaS.

There’s another thread about the same problem.

1 Like
5

Delation is very a strange thing for Camunda since more and more people trying to use Camunda as a gateway between frontend and backend to build low-code system

6

I use Self-Managed instance. Will installing my own exporter solve my problem? If I understand correctly by getting data directly from exporter I could get data immediately? I’m gonna use it for customer-facing tasks so any delations are not welcome

7

Also is it necessary to install my own exporter? What if I use default exporter Elasticsearch?

8

Hi @kristoffer.jalen , I am curious about your use case.
Do you want to build end-user facing applications?

Are the user who starts a process and sees the form the same?

For the case to simply show a form to a user, you can use a start form instead: Publish processes via a form | Camunda 8 Docs

You can also read them programmatically using the Web Modeler API.

To quickly show a task after another, it is recommended to use an exporter that listens to exactly that task.

Note that Camunda 8 doesn’t work transactional, instead it is event-driven with eventual consistency, so some traditional methods do not work well here. I acknowledge that our documentation might not yield this, and we will rework the docs.

Best,
Christian

9

Hi @kristoffer.jalen and @MagzhanUnited - Camunda is absolutely appropriate for customer-facing tasks, and many of our customers have processes that involved customers completing forms.

To add on to what Christian said, I would also suggest you might be solving the problem in the wrong way. First, Zeebe is asynchronous, so trying to wire up synchronous events can be difficult and may require a different solution. Second, having your application reach out to process means your application is more tightly coupled with the process. In one sense, your application shouldn’t care what step of the process you are on, because that is managed by the process engine.

Here’s a couple thoughts/questions:

  1. I know that during the development and testing cycle, a few seconds can feel like an eternity, but how often is that delay actually noticable to users of your application? How often are users aware that the previous step of the process had just completed and are staring at their computer screen waiting for a form to appear? (Of course I do not mean to dismiss the delay, which is long: more on that below.)
  2. I would recommend not using a User Task, and instead use a Service Task with either a job worker or Connector to push the task to your application, rather than your application trying to pull it. This has two benefits: first, the delay on the exporter is not a factor; and second, your application can remain further decoupled from the running process.

And again, 7-10 seconds is quite long; I experience a delay of 500ms-1s at most in my testing. @kristoffer.jalen - if you are willing, can you private message me the email for your SaaS account and/or the UUID of your organization (in the URL right after logging in)? I can check your cluster and discuss delay with our internal teams to get some further clarification there. (@MagzhanUnited - since you are using Self-Managed, the delay is much harder to identify because it depends on the environment you deployed it to, what resources are allocated to the services, etc.)

10

I use Camunda as a gateway between frontend and backend. A user starts process by clicking a button in the frontend then camunda makes request to the backend to get some data for form and returns form back to the frontend. But in my case there’s a hundreds of users in a website so every time when users interact with frontend creating process is heavy operation for server. Maybe I have some gaps in my knowledge about Camunda. Please correct me if I’m in wrong way

11

@MagzhanUnited - can you share your process model? Is that the entirety of your process, or after the form is completed by the user are there more tasks in the process?

If that is the entirety of the process, Camunda probably isn’t the best tool for that (unless you’re planning to expand the process). Camunda is optimized for process orchestration across services, which are often far more complex than just 2-3 steps. If you are only fetching some data then returning a form, it is probably far more performant and efficient to have your backend handle that directly.

12

diagram_2.bpmn (11.4 KB)

13

Hello, i am facing the same issue. i have to wait about 6 to 10 seconds after starting a process

14

Hello, i am facing the same issue

15

I believe there remains some misunderstanding here. Camunda is a process orchestration platform, powered by a distributed, asynchronous workflow engine (Zeebe). It is not a routing engine.

Camunda is designed and optimized for complex and high-volume process automations across multiple internal and external services. To use @MagzhanUnited’s process as an example, that process is far more efficient as a switch statement in your own application. (This is an area where our documentation could definitely use some improvement; I’ve shared these questions with the team, and we’re going to come up with a plan to improve it - if you have further feedback on this aspect of the documentation, please message me, I’d love to hear it!)

I think this blog post by one of our co-founders, Bernd Rücker, helps explain where process orchestration is beneficial and where it is not the right solution:

Camunda probably isn’t the right fit if you want to click a button and display a form or two to your user. However, imagine a loan application at a bank: once the application form is submitted, a larger series of tasks and decisions is initiated. You must validate the application, validate the applicant’s identity, pull their credit report, perform a risk assessment, and, depending on the loan type, many other steps, before the funds are disbursed to the account. Automating the loan application process end to end is a great use of Camunda and process orchestration.

Camunda enables creating and managing these processes, and what the individual tasks in each process do, with a no/low-code approach, a pro-code approach, or a mix. However, Camunda is not a drop-in no-code solution for your backend.

Now let’s talk about that “delay” and some solutions …

Depending on the size of your cluster in SaaS (or the resources you’ve allocated to your Self-Managed deployment), the speed at which a process instance starts is somewhat variable. If the first task in your process is a form, a delay of a few seconds from sending the request to start the process to the time that form is available in Tasklist or via the API is somewhat expected. Camunda is capable of running millions of process instances at a single time, so there are optimizations that can be done to help minimize that time delay. But remember that Zeebe is an asychronous and event-driven engine, not a synchronous and transactional one.

If a user interaction requiring a form is the start of your process, you can skip the process instantiation step entirely and start a process directly with a form.

If you have a more complex process that requires forms or human interaction in the middle (and you aren’t using Tasklist), I would consider using the “push” method I described in my previous reply: instead of your application polling the process to watch for these events, have the process tell your application when those events happen.

A similar “push” method can be used for events beyond user tasks; for instance, if your application needs to know when a process has ended, you can include a final service task right before the end event that notifies your application. (Generally speaking, it is a best practice to explicitly model as much of this as possible.)

Phew, that was a lot of writing!

Thanks for reading through this long reply! I felt it needed a much more detailed explanation, as we have received this same question from several users this week. Hopefully this helped clarify things; if not, please let me know what other questions you have. As mentioned earlier, this is something we need to improve in our documentation and I’ll be working with the team to make those improvements. If you have any specific feedback or thoughts on the documentation, please send me a message, I would love to hear it

4 Likes
16

Still facing with that issue. I create process using Zeebe client in nodejs then get active tasks directly from elasticsearch:

async activatedTasks(processInstanceKey: string) {
    const url = "http://localhost:9200/tasklist-task-8.4.0_/_search";

    // Define the query
    const query = {
      query: {
        bool: {
          must: [{ match: { processInstanceId: processInstanceKey } }],
        },
      },
    };

    // Send the POST request using Axios
    const response = await axios.post(url, query, {
      headers: {
        "Content-Type": "application/json",
      },
    });
    return response.data;
  }

If I do request I get response with empty hits so I need to delay code for about 10 seconds or sometimes 5-6 seconds.

I can’t start process with a form as you mentioned since my form requires data from rest api. So I tried to export form directly from default exporter which is ElasticSearch

17

@MagzhanUnited - as explained before, that is expected, because the data needs to be processed by Zeebe, sent to the exporter, written to Elasticsearch and indexed. Camunda is an asynchronous and event-driven platform. The process you presented is not an ideal use case for Camunda and process orchestration; if you are simply trying to create a form wizard that displays different forms based on different data conditions, there tools that are much better suited to that. I also presented other possible solutions that work with the event-driven nature of Camunda.

Over the next several releases, there will be some features that will help with what you’re trying to achieve (specifically in the 8.6 release in October). However, you will still likely have issues trying to use the asynchronous engine synchronously for your front-end.

18

I wanted to do the same thing as here: https://youtu.be/HhxghzKMFBg?si=fMm2NvU1T_NQ0LN8
Also my aim isn’t just displaying form on certain conditions. User tasks have to wait for its assigning and execution. I’m sure that Camunda is what I need.

19

@MagzhanUnited - that demonstration is 5 years old, and is using an older version of Camunda 7. C7 was a synchronous engine that could be embedded within your application. C8 is a ground-up rewrite that uses a faster, more scalable engine that is now asynchronous and event-driven. That solution does not work the same way with C8, and some C7 patterns need to be re-designed for C8.

As mentioned several times, you have a few options:

  • Use an event-driven architecture style, and have the model push changes to your application. This bypasses the exporter, which is the primary cause for the “delay.”
  • Use a custom exporter that does the same thing: pushes events to your application.
  • The 8.6 (October '24) and later releases will include not only improvements to the overall speed at which the exports occur, but also some additional features that will help with getting a more real-time view of the process. However, because C8 will still be asynchronous and event-driven, it will still not be the same as what you’re expecting.
20

Hi there,

Firstly, I want to express my gratitude for your time. Secondly, I’m having difficulty understanding two main aspects:

  1. How can I personalize my own exporter?
  2. How do exporters contribute to minimizing delays?