Zeebe architecture reasoning

First of all thank you for a great product!
After half a year with Zeebe I came to architectural questions. I would greatly appreciate if someone from architecture team could help me understand.

Currently Zeebe broker’s architecture includes the own log-based streaming implementation with end-user features very similar to Kafka + Kafka Streams.
For me it looks putting too much heavy lifting into the project when there are ready-to-use and mature solution are available.

Using external streaming could greatly simplify Zeebe’s architecture:

  • no need to worry about replications sync (it’s already implemented in Kafka)
  • no need in exporters (events can be consumed directly from message broker)
  • no need in gateway - clients can communicate with brokers via messages

Other pros/cons:

  • Kafka seems a bit too heavy for such use case, but it is very customizable and mature, also includes Kafka Streams to handle state store and change,
  • Freeing contributors’ time from maintaining streaming-related part could speed-up product development in other parts.

Might be I’m missing something in overall understanding.

1 Like

Hey Jetdream!

Thanks for asking! We answered that question in a FAQ quite a while ago: https://zeebe.io/faq/#is-zeebe-built-on-kafka. Quoting here:

…Zeebe is not built on top of Kafka.

Why not? There are a handful of reasons, and here are two of the most relevant.

  • Keeping Zeebe lightweight and easy to adopt: In addition to horizontal scalability, one of the reasons we’re building Zeebe the way we are (paritions + replications vs. relational database) is to limit the number of components required to use Zeebe. Requiring Kafka (and therefore ZooKeeper) to run Zeebe would have been a lot to ask from potential users, especially for those who aren’t using Kafka already, and we didn’t want to introduce that barrier.

  • Balancing flexibility and control over Zeebe with efficiency and maintainability: Building an engine on top of Kafka would’ve required us to put a lot of faith into a big and complex product that we don’t have much (if any) control over. That said, it wouldn’t have made sense to build every single Zeebe component from scratch, and we pull in other awesome open-source frameworks and tools where it’s possible. Zeebe relies heavily on RocksDB and gRPC, and we’re also working on integrating Atomix to cover some core functionality. At least for now, we’ve found a balance that allows us to build exactly the Zeebe that we want to build and that we believe will solve our users’ problems in the long-term while saving time and effort with existing libs and frameworks.

Hope that helps?