First of all thank you for a great product!
After half a year with Zeebe I came to architectural questions. I would greatly appreciate if someone from architecture team could help me understand.
Currently Zeebe broker’s architecture includes the own log-based streaming implementation with end-user features very similar to Kafka + Kafka Streams.
For me it looks putting too much heavy lifting into the project when there are ready-to-use and mature solution are available.
Using external streaming could greatly simplify Zeebe’s architecture:
- no need to worry about replications sync (it’s already implemented in Kafka)
- no need in exporters (events can be consumed directly from message broker)
- no need in gateway - clients can communicate with brokers via messages
- Kafka seems a bit too heavy for such use case, but it is very customizable and mature, also includes Kafka Streams to handle state store and change,
- Freeing contributors’ time from maintaining streaming-related part could speed-up product development in other parts.
Might be I’m missing something in overall understanding.