Zeebe usecase / architecture question

Camunda 8 Topics Discussion & Questions
1

Hi,

I’ve been reading the getting-started guide / documentations. The guide mostly seems to focus on specific workflows being started by a client application.

Is it possible for workflows to be started based on a queue? (internal zeebe queue, or external).

I am designing a system whereby when data changes in one application, I would like to put a “Data has changed” message on a queue, and have several workflows pick this message up, and decide whether to process it, based on the contents of the message.

Is this an OK approach to take, or should I re-think how I might try to do this with Zebee?

Is there a tutorial for this usecase that I may not be aware of?

Thanks
Dec

2

Hi @dec totally, this is a common use case for almost every workflow solution. So Zeebe is not the exception. Usually you will need to build a small component that will consume the message from the queue (that depends on which queuing system are you using) and that will map which message is supposed to start a process. That component is usually the client application that you are mentioning.

I am currently working on a component to make this easier, and I would love to have your feedback on it. There is a blog post coming on probably next week with the first iteration.

1 Like
3

Hi @salaboy, Excellent answer thanks! That makes perfect sense!

Would you use a queue that is built in to Zeebe, or something external like RabbitMQ / Kafka?

1 Like
4

@dec something external, we don’t want to limit your options… Imagine that for a cloud native project the integrations with cloud provider specific technologies is a must.

5

My tests are with Kafka now… but RabbitMQ should work as well.

6

Hi @dec, in addition to the approach that @salaboy pointed out, you can also use a message start event and publish your event into Zeebe as a message.

See the docs on Message events: Zeebe | Camunda Cloud Docs

And this tutorial on message correlation:

1 Like
7

Hey @salaboy
I was wondering if this has been implemented or if there’s any update.

I’m working on a use case where we’e going to process batch payments, so around 10K-100K payments per batch. For each payment we have 3-4 workflows that are initiated. It’s very much possible some of the Zeebe worker will be a source of bottleneck and won’t allow all the workflows to complete at once. So, I was wondering if Zeebe can automatically start workflows by pulling messages from a Kafka topic at it’s own pace based on the resources allocated.

Also, if I initiate a large number of workflows and Zeebe is not able to start all of them at once, is there queuing system in place such that it starts those workflows as resources open up?

Thanks!

8

There are no new features in Zeebe for this.

How long do the batch payments take to complete?

You could control the parallelization by awaiting the outcome, and having a thread pool on the starting process.

Also, if I initiate a large number of workflows and Zeebe is not able to start all of them at once, is there queuing system in place such that it starts those workflows as resources open up?

No, there is not. You will get a GRPC Error 8: RESOURCE_EXHAUSTED response in the client when this happens. The Node client has a transparent exponential back-off retry feature that handles this.

9