Zeebe and ETL workflows

Camunda 8 Topics Discussion & Questions
1

Hi,

I wonder how does Zeebe relate to traditional ETL tools like Airflow, Luigi or Prefect?

Unlike Airflow or others, Zeebe does not seem to provide any built-in tools akin to Airflow operators nor can I find any best practices how to deal with things like that: easily encapsulated small operations. I suppose this is by design, and as far as I understand, Zeebe is supposed to only orchestrate workflows and leave the execution to the orchestrated applications.

I’m however wondering how would you do implement Airflow-like operations in Zeebe? Say you want to move files between S3 buckets and your server, or move data between databases. Is there any guidance for this use-case, or is Zeebe even a fit for that? I suppose I could implement all different operations by hand, but that sounds a bit laborious if you can get it “free” from somewhere else. Any tips?

Best regards,
Matias

2

Hi @massive, welcome to the Zeebe Nation!

At the moment there are no small workers like this, but they are easy enough to build. I can see these things being, for example, reusable npm modules (if Node is your thing).

Keep an eye out for the announcement coming soon about the cloud offering. This will have some aspects that will affect this (re)use case.

Zeebe is designed to be a general-purpose scalable orchestrator, so specialisation like will likely be in the community ecosystem.

You could be the one to build it. That’s how I became the maintainer of the TypeScript client. It’s such an early stage that you can have a disproportionate impact with contributions.

Josh