Zeebe internal flow

Camunda 8 Topics
1

Hi Team,
I just would like to understand the zeebe internal flow, I got few queries, can someone confirm the same,

  1. ZeebeClient request send
  2. Gateway accept the request and forward to any partition leader
  3. Partition leader first append the command in the log file & replicate into follower
  4. Stream processor process the command and respond
  5. updated event will be append in the log file & replicate into follower

Queries:

  1. If I configured my snapshot duration as 10m, Every 10 mints my RocksDB will take a snapshop of these log file
  2. When I try to fetch current status of the process, my request will fetch the latest state from the RocksDB snapshot or event log
  3. When I try to deploy my diagram how & when it will get stored into elastic search database ?
  4. When my exporter fetch the records, all the event logs will be deleted ?
2

To answer your questions:

  1. Yes - snapshots are incremental, so this is fairly cheap.
  2. It depends how you fetch the status of your process. If you fetch it through the Operate API (or the new, soon to be released Camunda REST API), then the latest state is fetched from your history storage. This is either Elasticsearch, or OpenSearch.
  3. When something is replicated to enough followers, it is processed (as you pointed out), but it’s also exported to your history storage, making it available for a longer (configurable) retention period.
  4. Not immediately, but sort of: when an entry on the log has been processed and exported, it can be deleted. There’s some logic around it tied to snapshots and segment size, but that’s implementation details. Just assume once it’s been processed and exported, it’s quite likely deleted.
1 Like
3

Thanks for your quick response @npepinpe

I have doubt in point 2 still,

  • When I try to hit below apis, from where the records will get fetch, Is this elasticsearch/snapshot/eventlog

https://jfk-1.operate.camunda.io/{clusterId}/v1/variables/search
https://jfk-1.tasklist.camunda.io/{clusterId}/v1/tasks/search

4

Hello @vinothkumar
When you use the Tasklist or Operate API to retrieve data, it fetches the data from Elasticsearch.

5

Thanks @Kaveri_Govindasamy . Okay. So always my above mentioned APIs are fetched from snapshots ?

In this case, I have a scenario where my workflow instance will take 3 months to get completed. So the whole 3 months my event data is present in the Snapshots or Event logs ?

And also I have a doubt the snapshots will created for each request sending from zeebeclient ?

6

Yes, data is removed from the history storage (e.g. ES) only if the associated process instance is completed, and it was completed at least 30 days ago (30 days being the default retention, but that’s configurable).

7

Sorry @npepinpe . My query is bit different, my bad. I may put in different way,

https://jfk-1.operate.camunda.io/{clusterId}/v1/variables/search
https://jfk-1.tasklist.camunda.io/{clusterId}/v1/tasks/search

  1. Above mentioned api will fetch from ES or Snapshot or Event Log
  2. I have 2 workflow, first one will take 20 days to complete, so when I am querying on 20th date, the above query records will fetch it from the ES or Snapshot or Event Log
  3. second workflow will take 60 days to get complete, so when I am querying on 50th date, the above query records will fetch it from the ES or Snapshot or Event Log
  4. Is snapshot created for the each request sending from the zeebe client ?
8

Hello @vinothkumar,

Hope the below answer will help you:

  1. Both APIs will fetch data from ES, and all read calls will always fetch from ES.
  2. It will be fetched from ES. Also, the data can be retained depending on the historical data archival policy from the date of process completion. These are the defaults and can be configured:
    Operate: 30 days
Optimize: 180 days
Tasklist: 30 days
Zeebe: 7 days
  3. Same as answer #2.
  4. No, a snapshot is created when any of the following conditions are met: every 5 minutes, event log exceeds 5MB size of data, or 5K event records. However, every event is written to the in-memory state and replicated to multiple brokers before it is committed.

For more details please refer the below reference.

9

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.