Create multiple instances and wait for their completion

[moritzs]

Hi!

I would like to start several process instances of a process model that only contains external service tasks and wait for all process instances to be completed. I use an Docker Container for the logic of deploying the process model and starting the instances and an Docker Container for the Gateway and the Brokers with the official Docker Hub Images.

How can I be sure that all my process instances have been processed? I have already tried .withResult(), but this handles one instance after another, which is not what I want. I actually want all instances to start in parallel and also be processed parallel and check whether the last instance has been completed.

As stated in the documentation, I have already tried the non-blocking method with futures (https://docs.camunda.io/docs/apis-tools/java-client-examples/process-instance-create-nonblocking/)

long instancesCreating = 0;

while (instancesCreating < numberOfInstances) {
    // this is non-blocking/async => returns a future
    final ZeebeFuture<ProcessInstanceEvent> future =
            client.newCreateInstanceCommand().bpmnProcessId(bpmnProcessId).latestVersion().send();

    // could put the future somewhere and eventually wait for its completion

    instancesCreating++;
}

I suspect there is a way to wait for the individual instances to finish, but I don’t know what it is yet.

I would be very grateful for any help!

[claudio_rio]

Any reason you have to do this programmatically instead of using multi-instance sub-process or call activity?

[moritzs]

I haven’t thought about that yet. I’m still quite new to the subject.

I initially looked into the benchmark process and am currently working with it. That’s why I wanted to keep my process as simple as the benchmark process.
I’m doing a little experiment for myself in which I start the engine, deploy a process and then start a certain number of instances on it. I then want to determine the time it takes to run through all instances and compare them later. The time at which the first instance starts and the time at which the last instance ends are therefore also relevant for me.
I have now tried it with the multi-instance sub-process and it works. However, I do not know if or how I can get the start time of the first instance and the end time of the last instance to finish. I also do not know if the setup with the multi-instance sub-process puts more strain on the engine than simply starting the individual process several times.
My setup is therefore minimal. I only start the Zeebe container and the container that contains my ‘experiment’, because I only want to concentrate on the Engine.

For example: In my experiment container, I want to deploy the process model ‘benchmark’ and start 500 process instances. After the Docker container has started with the engine, my experiment container begins deploying the process model and then starts 500 process instances. I receive a UNIX timestamp before the first instance and a UNIX timestamp after the 500th instance so that I can calculate the duration. So now I want to make sure that all of the 500 instances are finished.

This is how I start the instances at the moment. The amount of the instances is coming from an env-variable in the docker compose.

for (int i = 1; i <= numberOfInstances; i++) {

            startInstance(processId, i);

        }

private void startInstance(String processId, int instance){
        if(instance==1){
            Instant start = Instant.now();
            long startMicros = getMicros(start);
            System.out.println("Instance #" + instance + " STARTED - " + startMicros);
        }

        zeebeClient.newCreateInstanceCommand()
                .bpmnProcessId(processId)
                .latestVersion()
                .send()
                .join();

        if(instance==processInstances){
            Instant end = Instant.now();
            long endMicros = getMicros(end);
            System.out.println("Instance #" + instance + " ENDED - " + endMicros);
        }
    }

So far I just tried it programmatically…

[claudio_rio]

If your goal is to measure performance then I think starting the process programmatically is the way to go.

Because you can have instances terminating in random order, you probably will have to wait for the instances to finish and the query to get instance information. With the earliest start date and the latest end date you can calculate how long the whole batch took.

See the API to query process instance information.

Search process instances | Camunda 8 Docs

[moritzs]

Since I don’t let operate run, as far as I understand, I can’t use the API either. So I believe that it doesn’t work that way. Maybe I’ll have to restructure my process a little after all.

I thought of something like this:

What do you think?

[claudio_rio]

The diagram you proposed may functionally get to the same result, but as it seems that your intent is to measure performance of having 500 processes running at the same time, the outcome won’t be much relevant unless to have a ballpark.

Maybe you can try the ZeebeClient.newProcessInstanceQuery to retrieve process instance information as well. Something like this (to get all finished instances; you can add more conditions to the filter)

var response = client.newProcessInstanceQuery().filter(p -> p.finished(Boolean.TRUE)).send();
var result = response.join();
// Process result.items();

Edit: this method is experimental. Bummer

[claudio_rio]

Using Camunda Run 8.8 and the Camunda Client, you can get the timestamps required to calculate the total execution time:

I created a dummy process with a Version Tag = “Benchmark”

public static void main(String[] args) {
	
	try (final CamundaClient client = CamundaClient
			.newClientBuilder()
			.grpcAddress(URI.create(ZEEBE_GRPC_ADDRESS))
			.restAddress(URI.create(ZEEBE_REST_ADDRESS))
			.preferRestOverGrpc(false)
			.build()) {
					
		var response = client
			.newProcessInstanceSearchRequest()
			.filter(f -> f.processDefinitionVersionTag("Benchmark").state(ProcessInstanceState.COMPLETED))
			.send();
			
		var result = response.join();
		
		System.out.println(">>> Found " + result.items().size() + " instances.");

		result.items().stream().forEach(p -> {
			System.out.println(p.getProcessInstanceKey() + ": " + p.getStartDate() + " - " + p.getEndDate());
		});
		
		Optional<Instant> earliestStartDate = result.items().stream().map(p -> p.getStartDate().toInstant()).min(Comparator.comparingLong(Instant::toEpochMilli));
		Optional<Instant> latestEndDate = result.items().stream().map(p -> p.getEndDate().toInstant()).max(Comparator.comparingLong(Instant::toEpochMilli));
				
		System.out.println(">>> Earliest Start Date = " + earliestStartDate.get().toString() );
		System.out.println(">>> Latest End Date = " + latestEndDate.get().toString() );
							
	} catch (Exception e) {
		e.printStackTrace();
	}		
}

[moritzs]

Thank you for your help so far! I am currently trying out my experiment, running only the engine in version zeebe 8.6.9/8.7 with the corresponding Docker container (https://hub.docker.com/r/camunda/zeebe).
Accordingly, the Camunda client is not yet available in these versions. I may have to consider testing directly with 8.8 if there is no way to solve my problem in 8.6/8.7.

[claudio_rio]

No problem. If you are starting something from scratch I recommend to go straight to 8.8.

There is migration effort from 8.7 to 8.8 can you can save yourself from

[claudio_rio]

@moritzs , check the reply to my inquiry in this topic:

Are the query methods for process instance available on CamundaClient

Hi @claudio_rio, Based on your question, I can classify this as a Problem - the methods you’re trying to use are not behaving as expected. The short answer is: These methods are not fully operational in Camunda 8.7, and this is expected behavior. Current Status of Process Instance Query Methods The newProcessInstanceSearchRequest() and newProcessInstanceGetRequest() methods you’re trying to use are part of the alpha Query API in Camunda 8.7. Here’s what’s happening: 1. Alpha Feature Status T…