Zeebe for Datapipelines

How would you compare Zeebe to things like Celery http://docs.celeryproject.org/en/latest/index.html ?

Hi @ab2, good question. I’m not too familiar with Celery, but I skimmed the docs and can give some general input.

Zeebe was designed primarily for orchestrating business processes (workflows) that span multiple microservices.

To work with Zeebe, in a nutshell, you’ll:
• Define a workflow in BPMN (including an option to define a workflow visually using a desktop modeler)
• Build worker services to carry out tasks in the workflow (or, if the services already exist, embed a Zeebe client)
• Deploy the workflow to Zeebe and start creating workflow instances (if you’re orchestrating an e-commerce order fulfilment process, a single order would probably be a single workflow instance)
• Zeebe, similar to a task queue, will create and store “jobs” whenever a workflow instance advances to a certain step, and worker services poll Zeebe to request and work on those jobs; job creation is decoupled from work being done on the jobs

Of course, Zeebe is “just” a workflow engine at its core and as such could be applied beyond the “microservices orchestration” use case, but its feature set has microservices orchestration top of mind.

Some other attributes of Zeebe that might be relevant:
• Zeebe doesn’t use a “message broker” for communication; Zeebe relies on gRPC for client-server communication
• One nice thing about gRPC is that it’s easy to generate clients in programming languages that gRPC support (10 as of now). The Zeebe project “officially” includes Go and Java clients, and users have contributed C#, Ruby, and NodeJS clients, too. I’m working on a Python client with one of my colleagues but haven’t put it out there yet :wink:
• Zeebe is built with horizontal scalability in mind; we initially started working on it to address orchestration use cases that might require many thousands+ workflow instances started per second.
• Zeebe can play nicely with e.g. Apache Kafka, if you want to build Zeebe workflows that “react” to messages published to a Kafka topic. You don’t need to have Kafka to use Zeebe, though.

Hope this helps. I kept it pretty general here–if you have specific questions about Zeebe’s capabilities, let me know!