Best practices for updating central state from external task workers

Camunda 7 Topics Discussion & Questions
1

Hi,
In my embedded process engine in Spring Boot application I want to maintain my application state in the same database as the camunda tables to leverage the transaction over a single datasource.

However, because about 20.000 process instances start at approximately the same time and they invoke a REST service that cannot handle too many calls, I consider solving this with external task workers.

This would make the architecture more complex, as it in involves more processes to deploy and maintaine; and I need to provide and secure access to camunda’s REST API. This could be
done when necessary, but I wonder how to update the application state in the database of the main application from the external task workers.

Options I see:

  • Pass the result as process variables from the external task worker call.
    This is bit cumbersome, cause I need an extra activity to store the data in my application state. This because the end-execution listener does not seem to work for external tasks. So it clutters may process diagram. And it uses process variables to pass application state, which I think should not be used when not necessary.
  • Invoke a REST endpoint from the main application to store the state.
    This would make the architecture more complex and could lead to inconsistencies between process state and application state.
  • Connect to the same database and apply the update there directly.
    In this case we would use multiple data sources, which does not match our initial design choice of one data source. Also it would even mean that multiple application share the same schema which is not what we want.
    This scenario also could lead to inconsistencies between process state and application state.

Another option is to embed the external task workers in the central application, using a custom thread to fetchAndLock available tasks and then execute them using a dedicated task executor. This seems to resolve the issues mentioned above and would be secure. However it feels a bit too creative.

I do wonder how the other camundo’s are doing this and what the best practices for mutating central state (in the application that embeds the process engine) are.

Best regards,
Patrick

2