External task ids are mixed up between concurrent running processes

Maysa.S · July 20, 2020, 2:00pm

I have a script that run in SQL Server, which results in new records inserted into a table.
I have service broker enabled, which publishes notifications on the table inserts. Meanwhile I have a C# application that is subscribing to those evens; the application consumes the events and trigger one of 5 Camunda BPMN Workflows (depending on the data). The script might insert 1000s of records in less than a minute into the SQL table; as a result, I have many camunda process instances running at the same time.

The 5 Camunda BPMN workflows each has multiple external tasks.
To execute the external tasks I do fetch and lock and handel them by executing some external C# methods then complete the tasks.

The issue is in some cases the external tasks Ids (taskId = fetchedTask.Id) are being mixed up between concurrent running processes instances, some external tasks are being executed and completed by a different running process instance instead of the one that is meant to execute it. As a result, I get wrong outcomes.

To explain more with for example

One of the Camunda BPMN workflows is called ‘Activation’
Camunda BPMN Activation workflow gets invoked twice at the exact same time
We get Activation workflow Process instance 1 & Activation workflow Process instance 2 which are running concurrently.
Process instance 1 starts with different set of data to Process instance 2
Process instance 1 fetches and locks external task with topic ‘Age’ & id ‘verifyAge’ and taskid ‘6675f702-c805-11ea-bf81-005056bd1798’ and start executing the logic.
Process instance 2, at the same time start executing the exact same fetched task by process instance 1 i.e. '‘6675f702-c805-11ea-bf81-005056bd1798’ and completes it even though it is still locked by process instance 1.
Process instance 1 fails complting taskid ‘6675f702-c805-11ea-bf81-005056bd1798’ as it has been completed already by process instance 2.

It is weird and not sure how this is possible.

Have anyone else experienced something similar where the External task ids get mixed up between concurrent running processes?

How can I prevent this from happening?

Ingo_Richtsmeier · July 21, 2020, 7:52am

Hi @Maysa.S,

how many workers/clients do you run?

If you run more than one, do they use different workerIds?

Cheers, Ingo

Maysa.S · July 21, 2020, 1:59pm

Hi @Ingo_Richtsmeier

I use one worker per external task and I even changed the maximum tasks to fetch to one i.e. MaxTasks = 1.

Below is code snippet of the worker

        var fetchPYS = new FetchExternalTasks()
        {
            WorkerId = "calculatePYSStatus",
            MaxTasks = 1,
            Topics = new List<FetchExternalTaskTopic>()
            {
                new FetchExternalTaskTopic("CalculatePYSStatus", 10000){}
            }
        };

        var tasksPYS = new List<LockedExternalTask>();
        tasksPYS = await _camundaClient.ExternalTasks.FetchAndLock(fetchPYS);

        var completeTaskPYS = new CompleteExternalTask()
        {
            WorkerId = "calculatePYSStatus",
        };


        foreach (var fetchedTask in tasksPYS)
        {
            string taskIdPYS = fetchedTask.Id;

            // Business Logic .....

            _camundaClient.ExternalTasks[taskIdPYS].Complete(completeTaskPYS).Wait();

        }

Where am I going wrong? it does work perfectly when I send one record at a time to Camunda process engine. This issue happens when I have many concurrent process instances running at the same time.

Any thoughts, ideas, direction will be appreciated.
Regards
Maysa