Process stucked at the start after redeployment

[ironWil1]

Hi!

My configuration:
I use Camunda Platfrom 7.18 in my services. I have two Spring Boot microservices using one shared DB for Camunda. Both services configured as:

  default-serialization-format: "application/json"
  admin-user:
    id: some
    password: some
    firstName: some
  filter:
    create: All tasks
  database:
    schema-name: some
    table-prefix: some
    type: postgres
  job-execution:
    deployment-aware: true

In the 1st Spring Boot app I have 3 different Camunda processes (.bpmn files). In the 2nd app I have 1 Camunda process (we divided it from the first app).

The problem:
After redeploying one of my Spring Boot App without any changes in Camunda scheme(.bpmn) all processes related to that App stucked at the start and don’t start. We repair the problem by pushing any small change in the scheme (for example just changing name of some element).

Camunda logs:

{"@timestamp":"2023-07-06T09:57:32.303Z","@version":"1","message":"ENGINE-13005 Starting command -------------------- SetExecutionVariablesCmd ----------------------","logger_name":"org.camunda.bpm.engine.cmd","thread_name":"http-nio-8083-exec-6","level":"DEBUG","level_value":10000}
{"@timestamp":"2023-07-06T09:57:32.304Z","@version":"1","message":"ENGINE-13009 opening new command context","logger_name":"org.camunda.bpm.engine.cmd","thread_name":"http-nio-8083-exec-6","level":"DEBUG","level_value":10000}
{"@timestamp":"2023-07-06T09:57:32.309Z","@version":"1","message":"ENGINE-13011 closing existing command context","logger_name":"org.camunda.bpm.engine.cmd","thread_name":"http-nio-8083-exec-6","level":"DEBUG","level_value":10000}
{"@timestamp":"2023-07-06T09:57:32.335Z","@version":"1","message":"ENGINE-13006 Finishing command -------------------- SetExecutionVariablesCmd ----------------------","logger_name":"org.camunda.bpm.engine.cmd","thread_name":"http-nio-8083-exec-6","level":"DEBUG","level_value":10000}
{"@timestamp":"2023-07-06T09:57:32.336Z","@version":"1","message":"ENGINE-13005 Starting command -------------------- CorrelateMessageCmd ----------------------","logger_name":"org.camunda.bpm.engine.cmd","thread_name":"http-nio-8083-exec-6","level":"DEBUG","level_value":10000}
{"@timestamp":"2023-07-06T09:57:32.336Z","@version":"1","message":"ENGINE-13009 opening new command context","logger_name":"org.camunda.bpm.engine.cmd","thread_name":"http-nio-8083-exec-6","level":"DEBUG","level_value":10000}
{"@timestamp":"2023-07-06T09:57:32.342Z","@version":"1","message":"ENGINE-13011 closing existing command context","logger_name":"org.camunda.bpm.engine.cmd","thread_name":"http-nio-8083-exec-6","level":"DEBUG","level_value":10000}
{"@timestamp":"2023-07-06T09:57:32.342Z","@version":"1","message":"ENGINE-16004 Exception while closing command context: Cannot correlate message 'ACCOUNTS_RECEIVED': No process definition or execution matches the parameters","logger_name":"org.camunda.bpm.engine.context","thread_name":"http-nio-8083-exec-6","level":"ERROR","level_value":40000,"stack_trace":"org.camunda.bpm.engine.MismatchingMessageCorrelationException: Cannot correlate message 'ACCOUNTS_RECEIVED': No process definition or execution matches the parameters...

The error says about problem with mesage correlation, because we try correlate message (in Java code by createMessageCorrelation(String messageName).processInstanceId(processId).correlate(), but our process stucked at the start.

My investigation:
After such redeploying without any changes in Camunda .bpmn scheme in Cockpit I found that Definition Version, Definition ID, Deployment ID don’t change, for example:

Definition ID:
3237e5a7-1bf7-11ee-b4ce-2e8fe5abe2b2
Deployment ID:
31f81d83-1bf7-11ee-b4ce-2e8fe5abe2b2

But Process Instances ID principle creation changed after redeploying (and new processInstances don’t work), for example:

Process Instances ID (before redeploy):
5edc7da7-1c00-11ee-b4ce-2e8fe5abe2b2
d08a1864-1c01-11ee-b4ce-2e8fe5abe2b2

Process Instances ID (after redeploy, don’t work):
5bca1ae5-1c05-11ee-ae05-2e6e28a7f71d

So, It seems that after redeploy it was new Deployment ID on which depends creation of new Process Instances ID. But in the table act_re_deployment I don’t have any similar deployment ID.

Could somebody help me with the problem, I don’t have an idea what’s happen after redeploy and why my processes crashing. In accordance with The processes.xml Deployment Descriptor | docs.camunda.org after redeploying without changes in the scheme.bpmn new deployment should’t be created, but in my case it semms that deployment was created and than deleted.

[GotnOGuts]

@ironWil1

  job-execution:
    deployment-aware: true

Deployment aware is difficult.
The individual engine will not work correctly after a restart, as it will not be aware of the fact that it has any processes deployed to it. This is why you are able to repair the problem by pushing any small change. The deployment is then registered to that engine. See also Deployment-Aware Job Executor
and also
The Job Executor | docs.camunda.org

[ironWil1]

Hi,
Without deployment-aware: true we had problems with scheduled tasks (timer intermediate catch event) . Sometimes one app catch events from another app.

[Ingo_Richtsmeier]

Hi @ironWil1,

if you set deployment-aware: true you have to ensure that the deployment is done by the spring boot application at startup.

Hope this helps, Ingo

[ironWil1]

Hi @Ingo_Richtsmeier!

Yes, my Spring Boot App make deployment at the every startup.

[GotnOGuts]

After startup of your Spring Boot Application, you must REGISTER the existing BPMN deployments to that instance (as per the links posted above).

The registration allows the process engine to load additional Java Classes and resources from the process application when executing the processes. In contrast to the database deployment, which can be restored whenever the process engine is restarted, the registration of the process application is kept as in-memory state. This in-memory state is local to an individual cluster node, allowing us to undeploy or redeploy a process application on a particular cluster node without affecting the other nodes and without having to restart the process engine. If the Job Executor is deployment aware, job execution will also stop for jobs created by this process application. However, as a consequence, the registration also needs to be re-created when the application server is restarted.

This is why doing another deployment of your BPMN causes it to start working.

Process Instance IDs do not change across the life of the Process Instance.