enableDuplicateFiltering is not working

I am using the Camunda BPM Spring Boot Starter Webapp. The issue that I’m facing is that with every restart of the application, a new version of all my processes is being deployed, even though there hasn’t been a change in any of them. I’ve set the enableDuplicateFiltering flag on the deployment builder to true, but it isn’t making any difference. What am I missing?

`

	// Deploy process definition of all tenants
	for (TenantInfo tenant : tenantInfos) {
		DeploymentBuilder deploymentBuilder = repositoryService.createDeployment().enableDuplicateFiltering(true).tenantId(tenant.getTenantId());// PROCESS_DEFINITION_PATTERNS = processes/tenant/*.bpmn
		String[] resourcePatterns = StringUtils
				.commaDelimitedListToStringArray(tenant.getProcessDefinitionPattern(PROCESS_DEFINITION_PATTERNS));
		Set<Resource> resources = getDeploymentResources(resourcePatterns);
		for (Resource resource : resources) {
			deploymentBuilder.addInputStream(resource.getFilename(), resource.getInputStream());
		}
		deploymentBuilder.deploy();`

Hi @ak27,

do you generate your process models via Model API?

Cheers,
Tassilo

I don’t think so? I’m creating them using Camunda Modeler. I’m not familiar with the Model API.

@ak27 Are you using in memory database(H2)?

Are you seeing different versions of same bpmn model in cockpit upon every restart or same version but only bpmn is getting deployed every restart?

I’m using the same setup with spring boot and bpmn files and doesn’t get new versions even without setting enableDuplicateFiltering.

@bmaehr can you provide the jdbc url which you are using?

aravindhrs I was using H2 database. I just tried switching to a SQL Server database that is being used as our dev db. No change in end result.

There is no change in the models at all. But with every restart, I’m seeing a new version of each model. I see a new record for every tenant in ACT_RE_DEPLOYMENT and within cockpit, I can see the number of deployments go up on the dashboard. If I select any of the running processes, I can see on the top left that the process definition version gets incremented with every restart. As such, to locate any of my active process instances, I have to select the previous version that it had started running in. I’ve been on this issue for days and honestly, I’m at a complete loss. I feel like it is something silly that I might be overlooking, but I just don’t see what.

@ak27 you configured to deploy the processes through spring boot auto deployment feature?

How are you deploying the bpmn files? Rest API or Postman or JAva API?

Hello,

I have the same problem here. I am starting my application in a Spring Boot Context and my process definitions are deployed with version 1 (assuming I have a new MySQL database). I am trying to re-deploy with the Java API at runtime, when the process is changed with the Camunda Modeler. But changes are not registered, instead the version is incremented every time and the changes are not visible (for example the name of the process).

The following DeploymentBuilder method has no effect, either with true or false: https://docs.camunda.org/javadoc/camunda-bpm-platform/7.7/org/camunda/bpm/engine/repository/DeploymentBuilder.html#enableDuplicateFiltering(boolean)

What is the correct way to update/re-deploy processes at runtime with the Java API? I also tried the following in my application.yaml (https://docs.camunda.org/manual/7.8/user-guide/spring-boot-integration/configuration/#camunda-engine-properties) with “.deploy-changed-only: true” but it had no effect aswell.

Thx.

1 Like

Maybe this is easier to understand - the following is executed when my application is started.

org.camunda.bpm.container: ENGINE-08024 Found processes.xml file at …/target/classes/META-INF/processes.xml
org.camunda.bpm.container: ENGINE-08025 Detected empty processes.xml file, using default values
org.camunda.bpm.container: ENGINE-08023 Deployment summary for process archive ‘app’
process1.bpmn
process2.bpmn

I need to execute this behavior when my application is already running to update my processes. I do not want to restart my application whenever there is a change to a process. Is it possible?

I am not using the Spring Boot auto deployment feature. If I try to use it, it creates 2+1 records in ACT_RE_DEPLOYMENT on the first startup (1 for each of my two tenants, and a 3rd with the name ‘SpringAutoDeployment’). I do not make any changes in my BPM diagrams and then find that on subsequent startups, there are 2 records created in ACT_RE_DEPLOYMENT again, 1 for each of my tenants. The SpringAutoDeployment record does not get created again.

To answer your question, I am deploying using the Java API as detailed in my original post.

@ak27 According to this Post Duplicate Process Definition a new ID is generated and this is why the process is considered different, even though the process itself has not changed. Application configuration like “camunda.bpm.application.deploy-changed-only: true” has no effect. Aswell as “enableDuplicateFiltering(true)” or "enableDuplicateFiltering(“false”) with the Java API does nothing.

Can someone solve this?

1 Like

So I’ve been playing around with this some more. I completely took out the code that was doing the deployment using the Java API and enabled auto deployment, with deploy-changed-only set to true.

It only kind of worked. I no longer see deployments happening on every restart. This is good. But, two major problems still exist:

  1. The deployed process definitions don’t have a tenant ID set on them. I was setting the tenant ID on the deployments using the Java API (see OP). I can’t figure out how to do it without.
  2. If there is a change in any one of my BPM diagrams, it is re-deploying ALL of the process definitions again, not just the one that changed. I don’t know if this is by design, but seems counter-intuitive considering I’ve enabled deploy-changed-only.

So, sadly this approach did not do it for me. I need to be able to use the Java API, but as @xndrexs mentioned, this method just refuses to work. Currently, I’m trying to figure out a way to manually check if a process definition has changed, and to only add it as a resource to the deployment if it has. But no luck so far. It’s looking increasingly as though there is a bug in the Java API, because the enableDuplicateFiltering flag is doing zilch.

So here is my workaround.

  1. I am checking to see if, for the given tenant, a deployment already exists.
  2. If no deployment exists, I am creating a new one and adding all the tenant’s .bpmn files to it. This becomes version 1 of the process definitions.
  3. If previous deployments do exist for the tenant, I am checking to see if the resource’s (i.e. my .bpmn file) last modified date is later than the date of the latest deployment for the given tenant. If true, then I am re-deploying all the .bpmn files for that tenant. Thus all the process definitions for that tenant get updated to version 2. If false, then I do nothing.

The issue I see with this is that if a deployment was done after a .bpmn file was changed (which can happen in an agile development lifecycle), then this approach will fail. I’m still mulling over what can be done about this. And of course, it’s not a very clean approach at all.

After all this I am convinced that either the enableDuplicateFiltering flag is broken, or it’s not designed to be used the way I’m trying to use it. If it’s the latter, then I still have no idea how we’re supposed to be checking for duplicate resources when doing a deployment. Should I be creating an issue for this?

for (TenantInfo tenant : tenantInfos) {
// PROCESS_DEFINITION_PATTERNS = processes/tenant/*.bpmn
		String[] resourcePatterns = StringUtils
				.commaDelimitedListToStringArray(tenant.getProcessDefinitionPattern(PROCESS_DEFINITION_PATTERNS);
		Set<Resource> resources = getDeploymentResources(resourcePatterns);
		long existingDeploymentCount = repositoryService.createDeploymentQuery().tenantIdIn(tenant.getTenantId())
				.count();

		if (existingDeploymentCount == 0) {
			DeploymentBuilder deploymentBuilder = repositoryService.createDeployment()
					.enableDuplicateFiltering(true).tenantId(tenant.getTenantId());
			for (Resource resource : resources) {
				deploymentBuilder.addInputStream(resource.getFilename(), resource.getInputStream());
			}
			deploymentBuilder.deploy();
		} else {
			Deployment existingDeployment = repositoryService.createDeploymentQuery()
					.tenantIdIn(tenant.getTenantId()).orderByDeploymentTime().desc().list().get(0);
			DeploymentBuilder newDeploymentBuilder = repositoryService.createDeployment()
					.tenantId(tenant.getTenantId());
			if (resources.stream().anyMatch(resource -> {
				try {
					return resource.lastModified() > existingDeployment.getDeploymentTime().getTime();
				} catch (IOException e) {
					logger.error("Error deploying Camunda resources", e);
				}
				return false;
			})) {
				for (Resource resource : resources) {
					newDeploymentBuilder.addInputStream(resource.getFilename(), resource.getInputStream());
				}
				newDeploymentBuilder.deploy();
			}
		}
	}

@ak27 Thank you for sharing.

@aravindhrs @tasso94 Can we please have a defect created for this?

Solution (finally!)

Hi @tasso94

I have currently problem with exactly this case. As I can see

org.camunda.bpm.engine.impl.cmd.DeployCmd#resourcesDiffer

implementation compares two XML process definitions - existing, and new (potentially the same as existing). But when these XMLs are generated by Model API, all element ids are generated on the fly and are different for every single deploy of the same process definition - e.g.

<incoming>sequenceFlow_5727473e-da38-4075-a0ac-2167d2b60869</incoming>, <bpmndi:BPMNDiagram id="BPMNDiagram_316dd423-39b4-4eb3-a4aa-2dde2c43152a">, <bpmndi:BPMNShape bpmnElement="..." id="BPMNShape_99f63448-ae44-4d4a-a315-f42d253188a9">

Hence, every generated process definition XML by Model API is different from previous and is deployed every time.

Do you know how to handle it?
For example, is there any way to implement my own id value generation strategy for these elements, to be sure they will have the same value every time?

v. 7.11