Hello,
I have a REST API call that takes around 30 mins to respond. While going through the Camunda Blog, I found,
Async also comes in handy if you have longer-running computations and do not want to block the calling thread (eg. HTTP Thread) but instead want to delegate the heavy lifting to a background thread.
Is it possible to execute a long running API call using proper transaction boundaries?
@Ankita-Saxena1 yes correct. You need to configure external task patterns
Go through this link:
Read about ext-client and use cases, ExternalTaskService, from-push-to-pull-external-tasks-in-bpmn-processes, and external-service-task-with-golang examples.
You can find the external task client examples:
But can we do that without implementing external tasks? For example placing asynchronous before/after flags in a connector.
Yeah of course,
But these jobs are executed by the camunda job executors. So camunda thread pool is used to handle these jobs, and you should be careful to choosing this approach over external task pattern for long running processes, because threads and resources are blocked in the call stack until the execution completes. If you have many jobs created for long running processes, sometimes all threads are busy in processing the request and you might get thread pool resource exhausted exception.
So are you saying that I’ll be able to execute an API (that takes 30mins to respond) using a connector by configuring these flags?
Yes correct.
AFAIK, that would still be blocking. All it relates to is the threading within Camunda, by setting transaction boundaries. I.e. an activity that is async before will be executed in a new job ASAP, but the execution of the job will take place in a thread from the thread pool made available to Camunda and will block that thread during the execution.
If long-running execution is your default case, the external task pattern that @aravindhrs pointed to is what was added for that purpose, assuming your execution asynchronicity is mainly a technical one. If it takes so long for functional reasons, however, BPMN offers a combination of send and receive task (or the equivalent events) to cover that. Think of cases such as “we send a letter to the customer, then wait for a reply” or “we call a remote API to open an new bank account which may take a day or two”.
We are actually not able to control the behaviour of the API from our end, so we are not able to implement send/receive task.
execution of the job will take place in a thread from the thread pool made available to Camunda and will block that thread during the execution.
will that thread be engaged till the API call is complete? Frankly, I tried using async flags for the API connector but it kept timing out in 60 or 120 seconds, so I needed to know if we can handle these long calls using the existing design.
using connector are you calling any external systems rest/soap api’s?
Yes, I am calling other apis from other containers in the docker
for long running tasks, you can use send & receive task or related events, only if they able to publish or callback to your system to update the completion status of the request. Otherwise external task pattern is the one simple and scalable way of solving this problem.
you can embed the external task client of java version as starters in your springboot application and create workers to poll and complete the tasks from job table and it will commit the status to process engine.
Yes, if you only use async before on a service task, it will block/engage the thread for the entire time the invocation of the API lasts. If you’re not able to influence the API, then you’re more or less forced to use the external task pattern to prevent the threads in the engine itself from blocking.
Hi @aravindhrs & @tiesebarrell ,
I have a workflow where I am making multiple REST API calls to many micro services (10-15) micro services to complete the process. On a positive scenario, assume each micro service may take 100 to 500 milli seconds(without failures/retries). What is the recommended approach? As we discussed above @ asyncBefore/after is just to create the transaction boundaries so that we can persist the state to the database till that point. Though asyncBefore/after creates a new thread to run the next transaction but it is still blocked waiting for the response from the RESP API. What is the recommended approach for this scenario?
Using an external task for each of these API calls?
Using send/receive tasks for this use case?
Or using the asynchronous service task?
Thanks much in adavance.
Yes. It’s to define the custom savepoint for the transactions.
I would prefer below approach for better scaling.
Thanks @aravindhrs .
When to use Asynchronous Service Invocations( [Request / Acknowledge / Callback Pattern]) ? What’s the use case for this?
For long running jobs and better scalability use External task patterns.
Requests that needs to be executed asynchronously can be configured and executed using job executors by defining transaction boundaries. In this case, process engine’s job executor worker threads will process the work items. There are certain benefits using async boundaries like, can return the response earlier and the subsequent tasks will be executed asynchronously, better transaction failures handling, reduced execution time, etc.