Data For Zeebe

Camunda 8 Topics
1

Hi,

I am learning and experimenting with Camunda 8. I am running Zeebe instance as a self managed deployment.

Scenario is: After configuring and running Zeebe broker, Elastic search and Tasklist, I changed the configuration to use my custom exporter (Writing a Zeebe Exporter - Part One - Camunda ) to log the events and data to console. I started the Zeebe broker without elastic search configuration.

When I start the zeebe broker, I see a lot of data written on the console(as you can see below) which is related to the process instances and tasks. Where is this data coming from ? How did camunda fetch this information related to past process instances while the elastic is not available ?

2022-06-16 13:56:01.583 {“partitionId”:1,“value”:{“version”:1,“bpmnProcessId”:“abc_process”,“elementId”:“Activity_0uh7n7k”,“flowScopeKey”:2251799813688902,“bpmnElementType”:“USER_TASK”,“processDefinitionKey”:2251799813688899,“parentElementInstanceKey”:-1,“processInstanceKey”:2251799813688902,“parentProcessInstanceKey”:-1},“key”:2251799813688913,“timestamp”:1655358213961,“position”:7554,“valueType”:“PROCESS_INSTANCE”,“sourceRecordPosition”:7551,“intent”:“COMPLETE_ELEMENT”,“rejectionType”:“NULL_VAL”,“rejectionReason”:“”,“brokerVersion”:“8.0.0”,“recordType”:“COMMAND”}

2022-06-16 13:56:01.583 {“partitionId”:1,“value”:{“version”:1,“bpmnProcessId”:“abc_process”,“elementId”:“Activity_0uh7n7k”,“flowScopeKey”:2251799813688902,“bpmnElementType”:“USER_TASK”,“processDefinitionKey”:2251799813688899,“parentElementInstanceKey”:-1,“processInstanceKey”:2251799813688902,“parentProcessInstanceKey”:-1},“key”:2251799813688913,“timestamp”:1655358213964,“position”:7555,“valueType”:“PROCESS_INSTANCE”,“sourceRecordPosition”:7554,“intent”:“ELEMENT_COMPLETING”,“rejectionType”:“NULL_VAL”,“rejectionReason”:“”,“brokerVersion”:“8.0.0”,“recordType”:“EVENT”}

2022-06-16 13:56:01.583 {“partitionId”:1,“value”:{“version”:1,“bpmnProcessId”:“abc_process”,“elementId”:“Activity_0uh7n7k”,“flowScopeKey”:2251799813688902,“bpmnElementType”:“USER_TASK”,“processDefinitionKey”:2251799813688899,“parentElementInstanceKey”:-1,“processInstanceKey”:2251799813688902,“parentProcessInstanceKey”:-1},“key”:2251799813688913,“timestamp”:1655358213964,“position”:7556,“valueType”:“PROCESS_INSTANCE”,“sourceRecordPosition”:7554,“intent”:“ELEMENT_COMPLETED”,“rejectionType”:“NULL_VAL”,“rejectionReason”:“”,“brokerVersion”:“8.0.0”,“recordType”:“EVENT”}

2022-06-16 13:56:01.583 {“partitionId”:1,“value”:{“version”:1,“bpmnProcessId”:“abc_process”,“elementId”:“Flow_088p9py”,“flowScopeKey”:2251799813688902,“bpmnElementType”:“SEQUENCE_FLOW”,“processDefinitionKey”:2251799813688899,“parentElementInstanceKey”:-1,“processInstanceKey”:2251799813688902,“parentProcessInstanceKey”:-1},“key”:2251799813688916,“timestamp”:1655358213964,“position”:7557,“valueType”:“PROCESS_INSTANCE”,“sourceRecordPosition”:7554,“intent”:“SEQUENCE_FLOW_TAKEN”,“rejectionType”:“NULL_VAL”,“rejectionReason”:“”,“brokerVersion”:“8.0.0”,“recordType”:“EVENT”}

2022-06-16 13:56:01.583 {“partitionId”:1,“value”:{“version”:1,“bpmnProcessId”:“abc_process”,“elementId”:“Event_0tftiix”,“flowScopeKey”:2251799813688902,“bpmnElementType”:“END_EVENT”,“processDefinitionKey”:2251799813688899,“parentElementInstanceKey”:-1,“processInstanceKey”:2251799813688902,“parentProcessInstanceKey”:-1},“key”:2251799813688917,“timestamp”:1655358213964,“position”:7558,“valueType”:“PROCESS_INSTANCE”,“sourceRecordPosition”:7554,“intent”:“ACTIVATE_ELEMENT”,“rejectionType”:“NULL_VAL”,“rejectionReason”:“”,“brokerVersion”:“8.0.0”,“recordType”:“COMMAND”}

2022-06-16 13:56:01.583 {“partitionId”:1,“value”:{“version”:1,“bpmnProcessId”:“abc_process”,“elementId”:“Event_0tftiix”,“flowScopeKey”:2251799813688902,“bpmnElementType”:“END_EVENT”,“processDefinitionKey”:2251799813688899,“parentElementInstanceKey”:-1,“processInstanceKey”:2251799813688902,“parentProcessInstanceKey”:-1},“key”:2251799813688917,“timestamp”:1655358213967,“position”:7559,“valueType”:“PROCESS_INSTANCE”,“sourceRecordPosition”:7558,“intent”:“ELEMENT_ACTIVATING”,“rejectionType”:“NULL_VAL”,“rejectionReason”:“”,“brokerVersion”:“8.0.0”,“recordType”:“EVENT”}

2022-06-16 13:56:01.583 {“partitionId”:1,“value”:{“version”:1,“bpmnProcessId”:“abc_process”,“elementId”:“Event_0tftiix”,“flowScopeKey”:2251799813688902,“bpmnElementType”:“END_EVENT”,“processDefinitionKey”:2251799813688899,“parentElementInstanceKey”:-1,“processInstanceKey”:2251799813688902,“parentProcessInstanceKey”:-1},“key”:2251799813688917,“timestamp”:1655358213967,“position”:7560,“valueType”:“PROCESS_INSTANCE”,“sourceRecordPosition”:7558,“intent”:“ELEMENT_ACTIVATED”,“rejectionType”:“NULL_VAL”,“rejectionReason”:“”,“brokerVersion”:“8.0.0”,“recordType”:“EVENT”}

2022-06-16 13:56:01.584 {“partitionId”:1,“value”:{“version”:1,“bpmnProcessId”:“abc_process”,“elementId”:“Event_0tftiix”,“flowScopeKey”:2251799813688902,“bpmnElementType”:“END_EVENT”,“processDefinitionKey”:2251799813688899,“parentElementInstanceKey”:-1,“processInstanceKey”:2251799813688902,“parentProcessInstanceKey”:-1},“key”:2251799813688917,“timestamp”:1655358213967,“position”:7561,“valueType”:“PROCESS_INSTANCE”,“sourceRecordPosition”:7558,“intent”:“ELEMENT_COMPLETING”,“rejectionType”:“NULL_VAL”,“rejectionReason”:“”,“brokerVersion”:“8.0.0”,“recordType”:“EVENT”}

2022-06-16 13:56:01.584 {“partitionId”:1,“value”:{“version”:1,“bpmnProcessId”:“abc_process”,“elementId”:“Event_0tftiix”,“flowScopeKey”:2251799813688902,“bpmnElementType”:“END_EVENT”,“processDefinitionKey”:2251799813688899,“parentElementInstanceKey”:-1,“processInstanceKey”:2251799813688902,“parentProcessInstanceKey”:-1},“key”:2251799813688917,“timestamp”:1655358213967,“position”:7562,“valueType”:“PROCESS_INSTANCE”,“sourceRecordPosition”:7558,“intent”:“ELEMENT_COMPLETED”,“rejectionType”:“NULL_VAL”,“rejectionReason”:“”,“brokerVersion”:“8.0.0”,“recordType”:“EVENT”}

2022-06-16 13:56:01.584 {“partitionId”:1,“value”:{“version”:1,“bpmnProcessId”:“abc_process”,“elementId”:“abc_process”,“flowScopeKey”:-1,“bpmnElementType”:“PROCESS”,“processDefinitionKey”:2251799813688899,“parentElementInstanceKey”:-1,“processInstanceKey”:2251799813688902,“parentProcessInstanceKey”:-1},“key”:2251799813688902,“timestamp”:1655358213967,“position”:7563,“valueType”:“PROCESS_INSTANCE”,“sourceRecordPosition”:7558,“intent”:“COMPLETE_ELEMENT”,“rejectionType”:“NULL_VAL”,“rejectionReason”:“”,“brokerVersion”:“8.0.0”,“recordType”:“COMMAND”}

2022-06-16 13:56:01.584 {“partitionId”:1,“value”:{“version”:1,“bpmnProcessId”:“abc_process”,“elementId”:“abc_process”,“flowScopeKey”:-1,“bpmnElementType”:“PROCESS”,“processDefinitionKey”:2251799813688899,“parentElementInstanceKey”:-1,“processInstanceKey”:2251799813688902,“parentProcessInstanceKey”:-1},“key”:2251799813688902,“timestamp”:1655358213974,“position”:7564,“valueType”:“PROCESS_INSTANCE”,“sourceRecordPosition”:7563,“intent”:“ELEMENT_COMPLETING”,“rejectionType”:“NULL_VAL”,“rejectionReason”:“”,“brokerVersion”:“8.0.0”,“recordType”:“EVENT”}

2022-06-16 13:56:01.584 {“partitionId”:1,“value”:{“version”:1,“bpmnProcessId”:“abc_process”,“elementId”:“abc_process”,“flowScopeKey”:-1,“bpmnElementType”:“PROCESS”,“processDefinitionKey”:2251799813688899,“parentElementInstanceKey”:-1,“processInstanceKey”:2251799813688902,“parentProcessInstanceKey”:-1},“key”:2251799813688902,“timestamp”:1655358213974,“position”:7565,“valueType”:“PROCESS_INSTANCE”,“sourceRecordPosition”:7563,“intent”:“ELEMENT_COMPLETED”,“rejectionType”:“NULL_VAL”,“rejectionReason”:“”,“brokerVersion”:“8.0.0”,“recordType”:“EVENT”}

Thanks in advance,
Srihari.

2

Hi

AFAIK zeebe uses an event sourcing paradigm and writes events to a local append log. Thus what you are seeing is the stream of zeebe events.

Normally these would also be streamed to elasticsearch as a stream of historical events where they can subsequently be read by operate, tasklist, optimize etc…

Regards

Rob

1 Like
3

Thanks for your reply Rob.

In case of Camunda 7, if the data source is not configured, the platform itself cannot boot because it needs all the data to start the engine.

I was expecting a similar case for Camunda 8, but I was surprised to see that there were no issues in the zeebe startup (I had a dummy exporter configured to log the events on console). That means, elastic is only used to store the data that was exported by zeebe. In this scenario, where exactly is the data like process definitions etc stored ?

4

Hi,

Thats a great question and one I cant answer yet…Im assuming zeebe has a process & decision repository, I suspect its derived from deployment events, but its not an area im familiar with yet…

Regards

Rob

5

@davidgs can you please throw some light on this topic ? Thanks in advance…

6

HI @hari_kiran

Data is stored in the append-only log and the RocksDB.

Deployed process definitions are in the RocksDB. See this post for more details.

Josh

1 Like
7