Hi Community,
I have implemented an ElasticSearch-Importer in .NET Core based on the Kotlin-Implementation ZeeQS.
At the beginning I configured just one partition and implemented accordingly the Importprocess. A loop gets data from the Zeebe-ES-Index, map it to the custom DB-Structure step by step and stores finally the processed Position in the ImportConfiguration. With the next Loop the process starts with the last processed Positon:
Query (+RangeQuery: position > 123) (size=200, sort by position asc) from index (zeebe*)
For a multiple partition scenario I found the following approach: To ensure a steady import time, I can import Zeebe records by a `sequence` field · Issue #10568 · camunda/zeebe · GitHub
I think with the sequence property it means for each configured partition:
e.g. partition 1
Query (+RangeQuery: 2…1 < sequence <= 2…200) (size=200, sort by position asc) from index (zeebe*)
e.g. partition 2
Query (+RangeQuery: 4…1 < sequence <= 4…200) (size=200, sort by position asc) from index (zeebe*)
After each loop processed for a partition, I store the last sequence value or the position in my ImportConfiguration-Table.
Are there any other approaches, ideas etc.? I’m really interested in your feedbacks.