Camunda 7.5.3-ee Deployment Methodologies - Redeployment Failing

We are experiencing a problem deploying Camunda 7.5.3-ee processes created in the Eclipse Modeler Plugin (not the standalone Modeler application) within Eclipse Mars.

We deploy a process through the WildFly admin GUI and it works fine within Camunda. We update the code for the process (which includes custom Java classes) and attempt to redeploy it via the WildFly GUI admin console. The deployment fails and the current deployment is corrupted.

This pattern was seen in combinations of WildFly 8.2.1, WildFly 10.0.0, WildFly 10.1.0, and Tomcat 8, with both Camunda 7.5.3-ee and 7.6.0-alpha5-ee.

My current theory is that because you have dropped support for the Eclipse Modeler plugin and the packaging methods it uses, that redeployment through “management” interfaces on WildFly and Tomcat is not supported. We are testing the theory that copying the build’s war file to the “deployment” directory of the Java container might work.

I’ve also wondered why others haven’t complained about this and thus my theory above, namely that we’re doing something we shouldn’t be doing. We know this worked in Camunda 7.4.5-ee on WildFly 8.

The WildFly developers have seen a stack trace and feel that the problem is an application integration issue.

This is a critical issue for us, so any help would be appreciated. We must be able to redeploy without removing an existing process so that process start requests are not lost.

Thank you.

Hi @mppfor_manu,

Could you please share a complete stacktrace? Could you please share your bpmn xml?

BTW, I do not think that your issue is related to the (not supported) eclipse modeler.

Cheers,
Roman

My theory about dropping the war file into the deployment directory have also failed. The stack trace from the initial deployment and the attempted redeployment follows. The process is used is as follows:

  1. Downloaded Camunda 7.5.3-ee/WildFly 10.0.0 bundle from Camunda site to my Windows 7 Professional 64 Bit workstation (Java 1.8)

  2. Unzipped the distribution

  3. Modified the management and public bindings to 0.0.0.0 and created a WildFly admin user

  4. Started Camunda using the start-camunda script

  5. Built the “loan-approval” process in Eclipse Mars with the Camunda Modeler Plugin installed.

  6. Copied the .war file to the WildFly “deployments” directory. WildFly detected the deployment. The process showed up in the Camunda Cockpit.

  7. Started an instance of the loan-approval process

  8. Incremented the version in Maven and changed the label on the user task of the process to provide a clear “marker” of the different version. No other changes were made.

  9. Built the process, which produces a war file with a different filename

  10. Copied the war file to the deployments directory of WildFly. WildFly detects this and attempts to deployment the process. The deployment fails within WildFly as shown below.

  11. Checked Camunda Cockpit and we see that there is a second version present.

  12. Attempted to start another instance of the process, but nothing happens. The dialog window just sits there forever until you close it.

The only way to make this process usable is to completely remove it and any references to it within Camunda, and even that may not work.

STACK TRACE:

2016-10-21 10:15:23,062 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0027: Starting deployment of “loan-approval-0.1.5-SNAPSHOT.war” (runtime-name: “loan-approval-0.1.5-SNAPSHOT.war”)
2016-10-21 10:15:23,087 INFO [org.camunda.bpm.container.impl.jboss.deployment.processor.ProcessApplicationProcessor] (MSC service thread 1-1) Detected user-provided @ProcessApplication component with name ‘org.camunda.bpm.getstarted.loanapproval.LoanApprovalApplication’.
2016-10-21 10:15:23,161 INFO [org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService] (ServerService Thread Pool – 74) Deployment summary for process archive ‘loan-approval’ of process application ‘Loan Approval App’:

    loan-approval.png
    loan-approval.bpmn

2016-10-21 10:15:23,162 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool – 75) WFLYUT0021: Registered web context: /loan-approval-0.1.5-SNAPSHOT
2016-10-21 10:15:23,180 INFO [org.camunda.bpm.application] (ServerService Thread Pool – 74) ENGINE-07021 ProcessApplication ‘Loan Approval App’ registered for DB deployments [ce332476-9798-11e6-b7e0-7a1120524153]. Will execute process definitions

    approve-loan[version: 1, id: approve-loan:1:ce354759-9798-11e6-b7e0-7a1120524153]

Deployment does not provide any case definitions.
2016-10-21 10:15:23,233 INFO [org.jboss.as.server] (DeploymentScanner-threads - 2) WFLYSRV0010: Deployed “loan-approval-0.1.5-SNAPSHOT.war” (runtime-name : “loan-approval-0.1.5-SNAPSHOT.war”)
2016-10-21 10:15:54,990 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (default task-52) RESTEASY002225: Deploying javax.ws.rs.core.Application: class org.camunda.bpm.tasklist.impl.web.TasklistApplication
2016-10-21 10:15:54,991 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (default task-52) RESTEASY002205: Adding provider class com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider from Application class org.camunda.bpm.tasklist.impl.web.TasklistApplication
2016-10-21 10:15:54,991 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (default task-52) RESTEASY002205: Adding provider class org.camunda.bpm.engine.rest.exception.RestExceptionHandler from Application class org.camunda.bpm.tasklist.impl.web.TasklistApplication
2016-10-21 10:15:54,992 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (default task-52) RESTEASY002200: Adding class resource org.camunda.bpm.tasklist.impl.plugin.resources.TasklistPluginsRootResource from Application class org.camunda.bpm.tasklist.impl.web.TasklistApplication
2016-10-21 10:15:54,992 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (default task-52) RESTEASY002205: Adding provider class org.camunda.bpm.engine.rest.mapper.JacksonConfigurator from Application class org.camunda.bpm.tasklist.impl.web.TasklistApplication
2016-10-21 10:15:54,992 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (default task-52) RESTEASY002205: Adding provider class org.camunda.bpm.engine.rest.exception.ExceptionHandler from Application class org.camunda.bpm.tasklist.impl.web.TasklistApplication
2016-10-21 10:18:03,470 INFO [org.jboss.as.repository] (DeploymentScanner-threads - 1) WFLYDR0001: Content added at location C:\camunda-bpm-ee-wildfly10-7.5.3-ee_3\server\wildfly-10.0.0.Final\standalone\data\content\7e\879ebadf391416a108f9525f4f55c114759928\content
2016-10-21 10:18:03,472 INFO [org.jboss.as.server.deployment] (MSC service thread 1-7) WFLYSRV0027: Starting deployment of “loan-approval-0.1.6-SNAPSHOT.war” (runtime-name: “loan-approval-0.1.6-SNAPSHOT.war”)
2016-10-21 10:18:03,497 INFO [org.camunda.bpm.container.impl.jboss.deployment.processor.ProcessApplicationProcessor] (MSC service thread 1-3) Detected user-provided @ProcessApplication component with name ‘org.camunda.bpm.getstarted.loanapproval.LoanApprovalApplication’.
2016-10-21 10:18:03,559 INFO [org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService] (ServerService Thread Pool – 79) Deployment summary for process archive ‘loan-approval’ of process application ‘Loan Approval App’:

    loan-approval.png
    loan-approval.bpmn

2016-10-21 10:18:03,559 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool – 80) WFLYUT0021: Registered web context: /loan-approval-0.1.6-SNAPSHOT
2016-10-21 10:18:03,571 INFO [org.camunda.bpm.application] (ServerService Thread Pool – 79) ENGINE-07021 ProcessApplication ‘Loan Approval App’ registered for DB deployments [ce332476-9798-11e6-b7e0-7a1120524153, 2dcda145-9799-11e6-b7e0-7a1120524153]. Will execute process definitions

    approve-loan[version: 1, id: approve-loan:1:ce354759-9798-11e6-b7e0-7a1120524153]
    approve-loan[version: 2, id: approve-loan:2:2dceb2b8-9799-11e6-b7e0-7a1120524153]

Deployment does not provide any case definitions.
2016-10-21 10:18:03,579 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-1) MSC000001: Failed to start service org.camunda.bpm.platform.process-application-module.“deployment.loan-approval-0.1.6-SNAPSHOT.war:main”.START: org.jboss.msc.service.StartException in service org.camunda.bpm.platform.process-application-module.“deployment.loan-approval-0.1.6-SNAPSHOT.war:main”.START: org.jboss.msc.service.DuplicateServiceException: Service org.camunda.bpm.platform.process-application.“Loan Approval App” is already registered
at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationStartService.start(ProcessApplicationStartService.java:162)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.jboss.msc.service.DuplicateServiceException: Service org.camunda.bpm.platform.process-application.“Loan Approval App” is already registered
at org.jboss.msc.service.ServiceRegistrationImpl.setInstance(ServiceRegistrationImpl.java:158)
at org.jboss.msc.service.ServiceControllerImpl.startInstallation(ServiceControllerImpl.java:235)
at org.jboss.msc.service.ServiceContainerImpl.install(ServiceContainerImpl.java:768)
at org.jboss.msc.service.ServiceTargetImpl.install(ServiceTargetImpl.java:223)
at org.jboss.msc.service.ServiceControllerImpl$ChildServiceTarget.install(ServiceControllerImpl.java:2401)
at org.jboss.msc.service.ServiceBuilderImpl.install(ServiceBuilderImpl.java:317)
at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationStartService.start(ProcessApplicationStartService.java:156)
… 5 more

2016-10-21 10:18:03,581 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 1) WFLYCTL0013: Operation (“deploy”) failed - address: ([(“deployment” => “loan-approval-0.1.6-SNAPSHOT.war”)]) - failure description: {“WFLYCTL0080: Failed services” => {“org.camunda.bpm.platform.process-application-module.“deployment.loan-approval-0.1.6-SNAPSHOT.war:main”.START” => “org.jboss.msc.service.StartException in service org.camunda.bpm.platform.process-application-module.“deployment.loan-approval-0.1.6-SNAPSHOT.war:main”.START: org.jboss.msc.service.DuplicateServiceException: Service org.camunda.bpm.platform.process-application.“Loan Approval App” is already registered
Caused by: org.jboss.msc.service.DuplicateServiceException: Service org.camunda.bpm.platform.process-application.“Loan Approval App” is already registered”}}
2016-10-21 10:18:03,603 INFO [org.jboss.as.server] (DeploymentScanner-threads - 1) WFLYSRV0010: Deployed “loan-approval-0.1.6-SNAPSHOT.war” (runtime-name : “loan-approval-0.1.6-SNAPSHOT.war”)
2016-10-21 10:18:03,603 INFO [org.jboss.as.controller] (DeploymentScanner-threads - 1) WFLYCTL0183: Service status report
WFLYCTL0186: Services which failed to start: service org.camunda.bpm.platform.process-application-module.“deployment.loan-approval-0.1.6-SNAPSHOT.war:main”.START: org.jboss.msc.service.StartException in service org.camunda.bpm.platform.process-application-module.“deployment.loan-approval-0.1.6-SNAPSHOT.war:main”.START: org.jboss.msc.service.DuplicateServiceException: Service org.camunda.bpm.platform.process-application.“Loan Approval App” is already registered

I followed the steps in the Java EE 7 steps to create the pizza-order process. I followed the same deployment process as before and it there were no issues. However, Camunda didn’t see the updates as being a different version.

I thought perhaps the issue in the loan-approval process deployment was the lack of an emptry beans.xml fileand persistence.xml file. I added those files to the loan-approval process and followed the deployment process described above. It failed, but there was a corrupted version 2 process definition. One other difference is that the build process creates a file whose filename includes the version. This filename changed as I incremented the Maven pom () version. This was not the case for the pizza-order process, even if I incremented the version.

Here’s the BPMN XML:

<?xml version="1.0" encoding="UTF-8"?>

<bpmn:definitions xmlns:xsi=“http://www.w3.org/2001/XMLSchema-instance” xmlns:bpmn=“http://www.omg.org/spec/BPMN/20100524/MODEL” xmlns:bpmndi=“http://www.omg.org/spec/BPMN/20100524/DI” xmlns:camunda=“http://camunda.org/schema/1.0/bpmn” xmlns:dc=“http://www.omg.org/spec/DD/20100524/DC” xmlns:di=“http://www.omg.org/spec/DD/20100524/DI” xsi:schemaLocation=“http://www.omg.org/spec/BPMN/20100524/MODEL BPMN20.xsd” id=“Definitions_1” exporter=“camunda modeler” exporterVersion=“3.1.0.201606051328” targetNamespace=“http://bpmn.io/schema/bpmn”>
<bpmn:process id=“approve-loan” name=“Loan Approval” isExecutable=“true”>
<bpmn:startEvent id=“StartEvent_1” camunda:formKey=“embedded:app:forms/request-loan.html” name=“Loan Request Received”>
bpmn:outgoingSequenceFlow_0c4zr6d</bpmn:outgoing>
</bpmn:startEvent>
<bpmn:sequenceFlow id=“SequenceFlow_0c4zr6d” sourceRef=“StartEvent_1” targetRef=“UserTask_11fud4o”/>
<bpmn:userTask id=“UserTask_11fud4o” camunda:formKey=“embedded:app:forms/approve-loan.html” camunda:assignee=“john” name=“Approve Loan 0.0.2”>
bpmn:incomingSequenceFlow_0c4zr6d</bpmn:incoming>
bpmn:outgoingSequenceFlow_1vbokmm</bpmn:outgoing>
</bpmn:userTask>
<bpmn:endEvent id=“EndEvent_1i5bz86” name=“Loan Request Approved”>
bpmn:incomingSequenceFlow_18sw98m</bpmn:incoming>
</bpmn:endEvent>
<bpmn:sequenceFlow id=“SequenceFlow_1vbokmm” sourceRef=“UserTask_11fud4o” targetRef=“ServiceTask_0m5ho94”/>
<bpmn:sequenceFlow id=“SequenceFlow_18sw98m” sourceRef=“ServiceTask_0m5ho94” targetRef=“EndEvent_1i5bz86”/>
<bpmn:serviceTask id=“ServiceTask_0m5ho94” camunda:class=“org.camunda.bpm.getstarted.loanapproval.ProcessRequestDelegate” name=“Process Request”>
bpmn:incomingSequenceFlow_1vbokmm</bpmn:incoming>
bpmn:outgoingSequenceFlow_18sw98m</bpmn:outgoing>
</bpmn:serviceTask>
</bpmn:process>
<bpmndi:BPMNDiagram id=“BPMNDiagram_1”>
<bpmndi:BPMNPlane id=“BPMNPlane_1” bpmnElement=“approve-loan”>
<bpmndi:BPMNShape id="_BPMNShape_StartEvent_2" bpmnElement=“StartEvent_1”>
<dc:Bounds height=“36.0” width=“36.0” x=“173.0” y=“102.0”/>
</bpmndi:BPMNShape>
<bpmndi:BPMNEdge id=“SequenceFlow_0c4zr6d_di” bpmnElement=“SequenceFlow_0c4zr6d”>
<di:waypoint xsi:type=“dc:Point” x=“209.0” y=“120.0”/>
<di:waypoint xsi:type=“dc:Point” x=“278.0” y=“120.0”/>
bpmndi:BPMNLabel
<dc:Bounds height=“20.0” width=“90.0” x=“198.5” y=“110.0”/>
</bpmndi:BPMNLabel>
</bpmndi:BPMNEdge>
<bpmndi:BPMNShape id=“UserTask_11fud4o_di” bpmnElement=“UserTask_11fud4o”>
<dc:Bounds height=“80.0” width=“100.0” x=“278.0” y=“80.0”/>
</bpmndi:BPMNShape>
<bpmndi:BPMNShape id=“EndEvent_1i5bz86_di” bpmnElement=“EndEvent_1i5bz86”>
<dc:Bounds height=“36.0” width=“36.0” x=“615.0” y=“102.0”/>
bpmndi:BPMNLabel
<dc:Bounds height=“20.0” width=“90.0” x=“588.0” y=“138.0”/>
</bpmndi:BPMNLabel>
</bpmndi:BPMNShape>
<bpmndi:BPMNEdge id=“SequenceFlow_1vbokmm_di” bpmnElement=“SequenceFlow_1vbokmm”>
<di:waypoint xsi:type=“dc:Point” x=“378.0” y=“120.0”/>
<di:waypoint xsi:type=“dc:Point” x=“454.0” y=“120.0”/>
bpmndi:BPMNLabel
<dc:Bounds height=“20.0” width=“90.0” x=“451.5” y=“110.0”/>
</bpmndi:BPMNLabel>
</bpmndi:BPMNEdge>
<bpmndi:BPMNEdge id=“SequenceFlow_18sw98m_di” bpmnElement=“SequenceFlow_18sw98m”>
<di:waypoint xsi:type=“dc:Point” x=“554.0” y=“120.0”/>
<di:waypoint xsi:type=“dc:Point” x=“615.0” y=“120.0”/>
bpmndi:BPMNLabel
<dc:Bounds height=“20.0” width=“90.0” x=“539.5” y=“110.0”/>
</bpmndi:BPMNLabel>
</bpmndi:BPMNEdge>
<bpmndi:BPMNShape id=“ServiceTask_0m5ho94_di” bpmnElement=“ServiceTask_0m5ho94”>
<dc:Bounds height=“80.0” width=“100.0” x=“453.667” y=“80.0”/>
</bpmndi:BPMNShape>
</bpmndi:BPMNPlane>
</bpmndi:BPMNDiagram>
</bpmn:definitions>

I have additional information. I decided to see if the loan-approval code itself was at fault in redeployment, so I “merged” the loan-approval code into another version of the pizza-order code. Essentially, I copied the BPMN, Java, and HTML code into a copy of the pizza-order project. I had to remove the loan-approval LoanApprovalApplication.java code. I also adjusted project names, etc.

I built the project and deployed it by copying it into the deployments directory. Both processes showed up in Camunda. I was able to start instances of both processes successfully. I could complete the processes if I wanted to, but left them running.

I then modified the Maven of the project, updated labels in both BPMN processes to reflect a new version, and updated a logger output filed in the pizza-order OrderBusinessLogic.java file. I also commented out the attribute in the pom.xml file so that the resulting war file from the build would have the version number. Thus, WildFly would see it as two distinct deployment files.

I built and deployed the updated project. It was deployed successfully and Camunda saw the updated process instances. I was able to start and complete processes for both the loan-approval and pizza-order workflows. Moreover, Camunda now had a version 2 of both process definitions.

This last test is the behavior we expect from WildFly/Camunda. So, we had two original, Camunda produced example projects: loan-approval and pizza-order. The loan-approval process could not be updated, but the pizza-order process could. However, when updating the pizza-order process, Camunda failed to recognize that it had gotten a new version.

It may be that Camunda only recognizes a new version if the file name is different than one previously deployed. This would be just fine as we can increment the Maven number.

At this point, I’m not sure what the difference between the two is. I tried to build and deploy the loan-approval process without the LoanApprovalApplication.java file, but Camunda would not recognize the process when it was deployed and threw errors, even though WildFly says it was deployed and enabled. However, loan-approval inside the pizza-order project without the LoanApprovalApplication.java file deploys and executes just fine.

The beans.xml file in both the standalone projects as well as the combined project is just an empty file. The persistence.xml file is the same for all projects. The processes.xml file for the combined project contains a modified process archive name. The pizza-order project did follow the structure of the documentation for Java EE 7, while the original loan-approval project lacked some files and directories. So, I’ll keep fiddling with project structure to see if I can figure out what the magic formula is.

Any further assistance would be appreciated.

I think I may have found the answer and will provide a full explanation soon. I just don’t want anyone to keep looking for an answer.

In short, the project setup, inclusion of the JBoss EE dependency, and a couple of other possible factors appear to have been the culprits.

The problem, it appears, was the use of the “@ProcessApplication” Java annotation statement. Commenting this out seems to have resolved the problem with redeployment.

The use of this statement in our code as well as the loan-approval sample code is, I believe, a legacy of previous deployment configurations from earlier versions of Camunda. Our mistake was that we simply continued to create an application “implementation” file because that was part of the tutorial process a couple years ago. As such, we incorrectly assumed that it was just one of those artifacts that you had to have in your projects without really knowing or caring what it did.

Sadly, we still have a problem and I’ve no idea what to do about it. Removing the “@ProcessApplication” from the Java “implementation” class caused Camunda to ignore the deployment, even though it was successfully deployed to WildFly.

Following is source code from that file. I am not the developer, but my testing indicates that a class such as this isn’t necessary for Camunda to recognize a process. This is clearly something we are doing wrong, because the pizza-order example doesn’t have “@ProcessApplication” anywhere in the code. If you need other parts of the actual deployment, let me know and I’ll ask if I can send it (it contains proprietary code, so I would only be able to send it to a non-public, secure endpoint).

package com.att.gcs.bizops.cop.core.template.messagehandler;

import org.camunda.bpm.application.ProcessApplication;
import org.camunda.bpm.application.impl.ServletProcessApplication;
import org.camunda.bpm.engine.delegate.ExecutionListener;

@ProcessApplication(“MessageHandler”)
public class MessageHandlerApplication extends ServletProcessApplication{

public ExecutionListener getExecutionListener() {
	return new ActivityLoggingExecutionListener();
 }

}

Hi mmpfor_manu,

if you include the dependency to the camunda-ejb-client, you get a generic ProcessApplication class in your project.

    <dependency>
      <!-- client for Java EE application server integration, included in WAR as an alternative to write your own ProcessApplication class -->
      <groupId>org.camunda.bpm.javaee</groupId>
      <artifactId>camunda-ejb-client</artifactId>
    </dependency>

We use it in the maven archetype camunda-archetype-ejb-war-7.5.0.

Cheers, Ingo

I discovered that a couple of hours ago and was able to get things to deploy properly, but hadn’t updated this thread.

Thank you for your response as it would have solved my problem. At least until testing proves otherwise.

Michael

@Ingo_Richtsmeier

Hi, do you we have sample process with camunda-ejb-client, so that we can replicate our process to work.

we are seeing issues when we run our app with camunda-ejb-cleint.

Also do we have versioning using @ProcessApplication in camunda. Please suggest.

Thanks,
Sridhar

Hi Sridhar,

generate your project from the camunda-archetype-ejb-war-Archetype as described here: https://docs.camunda.org/manual/7.5/user-guide/process-applications/maven-archetypes/#detailed-instructions or by the command line: https://docs.camunda.org/manual/7.5/user-guide/process-applications/maven-archetypes/#usage-on-commandline

You will get a complete, deployable process application with a minimal executable process from it.

Cheers, Ingo

Please provide more details. Did you properly configure the MySQL database within whatever container you are using? Your application shouldn’t be affected by the Camunda database server if it was set up correctly. Be sure to execute the proper MySQL scripts located off the installation directory.

Michael

It depends upon which war file you are referring to. If it is one of the Camunda war files in the standard distribution, then that is a bit odd. If it is one that you created, then you should rebuild and redeploy the file. You might have to manually remove the current, corrupted copy from the webapps directory.

Michael