Process Deployments with failing startup of process application

Hello,

in my server I have two WAR files containing two processes of the same id.

myProcessV1.war

  • myProcessV1.bpmn (id: “myProcess”, isDeleteUponUndeploy=false)

myProcessV2.war

  • myProcessV2.bpmn (id: “myProcess”, isDeleteUponUndeploy=false)

When I start the engine, two deployments of “myProcess” are created. No problem so far. While myProcessV2.war is deploying, an error occurs in its process application class, which causes an undeployment of myProcessV2.war. At this time there are still two deployments of “myProcess” in the camunda database.

Problem:
With every restart of the described situation two deployments of “myProcess” are added in the camunda database.

What I expected
Even if the two deployments of “myProcess” are justified (although deploying the second process application failed to start), there must not be two new deployments after each server restart. The bpmn files did not change. Overall a maximum of two deployments are realistic here, independent from server restart count.

Is there something wrong with the deployment mechanism or with my understanding of the same?

Thanks in advance!

Best regards
Carsten

Edit:
The two bpmn-files must have the same name (e.g. “myProcess.bpmn”). This criterion is enough to reproduce the problem. The startup failure is not relevant.

Hi Carsten,

Versioning of processes in Camunda is based on the process key (aka id attribute of the process element in the BPMN 2.0 XML). Also, during deployment, Camunda only considers the latest version when deciding whether a process should be deployed. Second-latest versions, etc., are not considered.

In consequence, when you have applications with different versions of the same process (i.e. same key) deployed at the same time, the deployment behavior is undesired. Processes are getting deployed repeatedly on top of each other as you have observed. Thus, this situation should be avoided. Either make the keys distinct, or ensure that the deployed process applications always contain the same versions of the BPMN XML. If both is not possible for you, then please elaborate on your use case so I can understand it better.

Cheers,
Thorben

Hi Thorben,

I accept that it is not the preferred way, to contain same processes (with different versions) in two process applications (war-files). But nevertheless the behaviour seems strange to me. I´ll try to explain with other words.

  1. Step
    Deploying myProcessV1.war (with myProcess.bpmn)
    -> Deployment: myProcess [Version 1…]
    -> OK

  2. Step
    Adding myProcessV2.war (with myProcessV2.bpmn)
    -> Deployments: myProcess [Version 1…] + myProcess[Version 2…]
    -> OK, even if the web archive deployment of the new war file fails

  3. Step
    Restart server wih myProcessV1.war and myProcessV2 in the webapps folder. Nothing changed in the archives.
    -> Deployments: myProcess [Version 1…] + myProcess[Version 2…] + myProcess[Version 3…] + myProcess[Version 4…]

  4. Step
    Restart server wih myProcessV1.war and myProcessV2 in the webapps folder. Nothing changed in the archives.
    -> Deployments: myProcess [Version 1…] + myProcess[Version 2…] + myProcess[Version 3…] + myProcess[Version 4…] + myProcess[Version 5…] + myProcess[Version 6…]

and so on. This cannot be the expected behaviour. The bpmn files always stay the same. How does camunda detect if a bpmn file requires a new deployment or represents an already existing deployment? Isn´t the content the relevant criterion?

Best regards,
Carsten

Edit:
I do not have this behaviour if deploying the second process application (myProcessV2.war) works fine. Only the failure while deploying this process application seems to cause the problem/behaviour. I know that startup failure of a process application is an unacceptable situation and must be solved at first. But I think that camunda would have to behave in another way even if such an error occurs.

At which part during deployment does the exception occur? In an @PostDeploy annotated method of your process application or in another method?

The exception occurs because the two process applications have the same name.

EXCEPTION:org.camunda.bpm.container.impl.ContainerIntegrationLogger.exceptionWhilePerformingOperationStep(ContainerIntegrationLogger.java:312): org.camunda.bpm.engine.ProcessEngineException: ENGINE-08043 Exception while performing 'Deployment of Process Application Monitoring versions' => 'Start Process Application Service': Cannot register service org.camunda.bpm.platform.job-executor.process-application:type=Monitoring versions with MBeans Container, service with same name already registered.
 caused by org.camunda.bpm.container.impl.jmx.MBeanServiceContainer.startService(MBeanServiceContainer.java:68): org.camunda.bpm.engine.ProcessEngineException: Cannot register service org.camunda.bpm.platform.job-executor.process-application:type=Monitoring versions with MBeans Container, service with same name already registered.

Hmm, I can’t reproduce the behavior you describe. I use the Camunda 7.6.0 Tomcat distribution downloaded from camunda.org.

I have two artifacts v1.war and v2.war.

I use a servlet process application for both artifacts:

@ProcessApplication(name="servlet-pa")
public class ExampleProcessApplication extends ServletProcessApplication {
}

I use this processes.xml for both artifacts_

<?xml version="1.0" encoding="UTF-8" ?>
 
<process-application
    xmlns="http://www.camunda.org/schema/1.0/ProcessApplication"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

  <process-archive>
    <process-engine>default</process-engine>
    <properties>
      <property name="isDeleteUponUndeploy">false</property>
      <property name="isScanForProcessDefinitions">true</property>
    </properties>
  </process-archive>

</process-application>

In v1.war, I have a process model named exampleProcess.bpmn. In v2.war I have exampleProcess2.bpmn. Both processes have the same key but not the exact same content.

No matter how often I restart Tomcat, I end up with two deployments and process versions, not more.

How does your scenario differ?

Cheers,
Thorben

I can´t find any significant differences in your example. That´s exactly what I do. Does your v2.war fail at startup, too?

Best regards,
Carsten

Yes it does, resulting in the same error message. I assume you use 7.6.0 on Tomcat as well?

I am using camunda 7.6.0, but with a Tomee. Can that cause the problem?

I don’t know, we don’t test TomEE in our CI. You could try reproducing the issue on the regular Tomcat distribution yourself to check if TomEE or any configuration setting influences this.

Now I used the current tomcat distribution from camunda.org. I can reproduce the problem there, too. Unfortunately, I cannot upload my two war files here.

Do you mean this in terms of technical issues or that they are confidential?

No, they are not confidential. The upload mechanism bars me from uploading war-files.

Could you put the sources up on github?

I uploaded my sources here: https://github.com/SpiekerC/camunda-examples/tree/master/deployment-issue

I must apologize for missing one detail. I just saw that my bpmn-files had the same name. As far as I can see this is also an important criterion for reproducing the error.

Edit:
Root cause of the problem is NOT the error while startup, but the existence of two bpmn files with the same name (in two different web archives)

Hi Carsten,

in that case, this is indeed the expected behavior and should (as you have observed) behave the same way in both cases. The explanation is roughly the same as what I tried to explain in my first post, although I was not entirely correct.

When making a Camunda deployment, the process engine uses the following strategy (with default settings) to determine which resources to deploy:

  1. Collect resources contained in deployment. With classpath scanning enabled, this will include all .bpmn-suffixed files. Let’s call this set of resources R1.
  2. Collect the latest versions of resources of previous deployments with the same name from the database. A resource is the latest version if there is no other resource with the same name that belongs to a newer deployment. Let’s call this set of resources R2.
  3. For each resource in R1, check if there is a resource in R2 with the same name. If yes, make a binary comparison of both resources. Let’s call the set of changed resources R3.
  4. If R3 is not empty, then make a new deployment containing all resources in R1.

Now, if you have two process applications (A1, A2) with a resource with the same name, but different content (C1, C2), the following will happen:

  1. Deploy A1. Create a new deployment with resource content C1
  2. Deploy A2. Create a new deployment with resource content C2, because the latest resource with the same name has content C1 and C1 != C2.
  3. Restart the server. During startup, Camunda will go through the deployment routines of both A1 and A2. The order is undefined, but let’s assume A1 is considered first, and A2 is considered second (the other way round results in similar behavior). When attempting deployment of A1 with resource content C1, the latest deployed resource has content C2. Since, C1 != C2, this creates a new deployment. The same happens when A2 is processed next
  4. And so it repeats with every server startup.

In consequence, please avoid the situation where you have multiple process applications with deployments and resources of the same name, but different content. The versioning mechanism in Camunda does not work this constellation. It assumes that there is always one application that is responsible for the latest and previous versions of a deployment at a time.

I hope this makes sense.

Cheers,
Thorben

2 Likes

Hi Thorben,

yes that makes sense. The new and crucial information for me is that the resource name (in this case the defining bpmn file of a process with a given process id/key) is relevant for finding existing deployments for the resource in the database.

Thanks for your patient handling of my problem and the detailled explanation. We could have shortend it, if I mentionend the duplicate bpmn file name directly at the beginning. Sorry for that. :neutral_face:

Best regards,
Carsten