I try to find the right design for my micro service process.
I work for an IoT network provider. When a subscriber buy an IoT device, we need to configure it to be able to acces to the network. An order can have items (devices) from multiple suppliers.
We have two offerings for our IoT platform, on-premise and saas. The on-premise platform can access to the Saas Suppliers.
We already have a process definition (Activiti engine), it’s a big monolith with two phase commit. According to the Bernd Ruecker blogs, github projects, i’m currently architecturing the migration to a micro service process and saga pattern.
The blue lanes are for normal processing and the red one are for compensation.
The order process call the Supplier micro service (local) to configure goods (AMQP / Kafka).
The Supplier micro service launch a process to configure all goods (devices) by Supplier. Like some supplier can be from Saas, some call are done by AMQP/Kafka and others by Rest call. The process check if all Supplier are all right. If not, a compensation message is send to the Order process.
The supplier micro service (local, saas), call its supply chain to configure the device. It send a success or failure message response.
The compensation mecanim is globally the same things.
It’s a valid / good / best way to have micro service + compensation for multi instance call ?
We’ve been working with a lot with companies migrating to a more micro-service oriented architecture. Which is why Bernd has had the opportunity to write so well about it. The fun thing is that i don’t think I’ve come across anything I’d consider the “best” way of doing it. Each individual project has it’s own restrictions, requirements and goals so i wouldn’t want to suggest a one-size-fits-all approach.
With what you’ve posted so far - it seems like a perfectly reasonable approach (your approach to message correlation is going to be interesting).
If @BerndRuecker wants to add his two cents to this you might get a more educated opinion
I aware there is no general response. The things that intrigues me is the multi-instances call’s to micro service with asynchronous call / response and compensation like in my other post. I think for this case a general approch is possible, i trying to find how to do it. Plus like explained, i’m wondering if i need that all multi instance calls be must completed before compensate or i can do it only for completed (transaction + cancellation).
Hi Jerome.
Sorry for jumping in the discussion so late!
One question about how you see your business domain: is a supllier/goods microservice responsible for one good or for the whole bunch contained in an order? Naturally I would expect that the order domains split it up to multiple calls to the supplier domain, moving the multiple instance into the order flow. But that’s just a gut feeling - which is something I shouldn’t do without having any details
But anyhow, I could imagine it that it should look a bit different, so either:
A) Having the multiple instance in order directly - meaning that you call supplier mutliple times. In this case you need to have compensation for the individual calls (probably involving a multi instance sub process).
B) Having the compensation directly in the supplier (if you work on a bunch of items at once - as soon as one fails you can do compensation already locally in the supplier service - or not?). Then the order process doesn’t have to do compensation for goods at all.
My first design used a multi instance on the Order process. Both design go well for my use case.
A), if the multi instance is on the order, i’m agree that i need a compensation for each instance. I failed to find the good design for Saga pattern, compensation and multi instance sub process with camunda. I need to compensate all completed instances and to stop the others one or wait that all completed and compensate the all. But i don’t want to introduce some god object that track the item’s state in the multi instance sub process or add a task that check all done with DB call (like in my pevious design) before compensate. I’m convinced the BPMN standard provide usefull tool like the compensation mechanism to do it, but i need to find how to apply it to multi instances sub process in order to propagate the compensation to the tasks before, in the reverse order. It’s the normal behavior for Saga pattern. I’ve not succeed to do that with camunda.
B), my requirement is to compensate all if any micro service (goods or others) failed. Compensation in the good micro service won’t be usefull because the main process, in orchestration scenario (order process) must be aware of that failure to compensate others instances and previous tasks.
@BerndRuecker have you already facing this kind of design (multi instance + compensation) and find a way to use the Saga pattern?
I did a couple of experiments and can confirm that currently you need some workarounds to get your situation around MI and compensation working correctly. If you need a solution now I think using some workaround as Niall described is best. As written in another thread: We will have a look at MI + Compensation in upcoming 7.9 - but I cannot yet promise anything.
Thank you for you reply. The Niall workaround are not working for our uses cases.
The first workaround that fit some of our requirements have been found yesterday. We use an integration framework, Camel, to send all our requests to micro services in the MI sub process. The camel route + pipeline aggregate all responses and after all responses have been received, trigger a global event subprocess with compensation if one of the request / response failed or trigger a success message that is waiting after the MI sub process. All technicals details are done in the integration framework and don’t pollute the process diagram.
This workaround have sone side effect that need to be fix.
The first one found today is that all MI sub process must succeed without error (other that the micro service response). Indeed, if some error occur in an instance of the sub process, compensation will be done automatically inside the MI sub process and the default camunda behaviour will be trigger (not good).