Just adding read-only queries on the API need to change the architecture ? I must have miss something here…
Thanks! Glad to here that! But I just want to discuss here about my thoughts before adding any Github issues. I want to be sure that my needs can’t be fulfilled without new developments.
I prepare a Proof of Concept project around Zeebe so I need to known a lot about Zeebe architecture and philosophy before validating the concept. What is working well, and what is not, what is possible, and what is not possible.
I have already explain a problem I am facing right now, where I have deleted exported data in Elasticsearch, keeping Zeebe data like precious ones.
I was thinking that deleting Elasticsearch data will “reset” Zeebe (I have read this silly advice in Github issues, but maybe I misread), but no, Zeebe store more than cluster information on disk, it also have its RocksDB filled with event logs, allowing to track state of workflows/instances/etc…
If Zeebe DB is the critical core data storing the state of all my micro-services orchestration, you can understand that I need to have an complete access to it. Events sent to exporters are not relevant to me as it is only a copy of the data and can be corrupted for whatever reason or just have a rolling delete to avoid flooding.
From a security backup point of view, I need to know where is the critical data, the one that allow me to recover and restart the system exactly from the state I was before an incident.
I can not sell a service to a customer where the critical data is a black box that I can not analyze/debug/save/restore/recover/repair.
I have read all the Basics chapter (at last!) and have a better understanding of the internals of Zeebe.
To achieve a resync (get Zeebe to export current state to exporters), may be we can use the snapshot functionality of Zeebe.
Extract from the blog:
As the snapshots are used during recovery, its imperative that they are made durable, and are correctly replicated. The team therefore introduced checksum validation during snapshot replication, and improved the durability of snapshots on creation as well as during replication.
I see two possibilities here:
- Send an API command to get or (better) send to exporters all the logs of the Zeebe partition.
- Send an API command to get or send to exporters the snapshot of the Zeebe partition.
Doing that, we respect Zeebe philosophy:
Send command via API -> store command event -> execute command -> send response to API and/or Export data to exporters.
Do you think this is realistic ?