Unexpected error occurred during RocksDB transaction.

I am a new user of Camunda. After deploying Zeebe to AWS using Fargate and installing the Hazelcast exporter to use the Simple Zeebe Monitor, I mounted EFS to the zeebe_data folder. However, I am encountering an error, and no job worker can connect to Zeebe.

Could anyone please help me understand why this is happening and how to prevent the error from occurring again?

Thank you for your assistance.

2024-11-29T13:36:09.672Z
io.camunda.zeebe.broker.exporter.elasticsearch - Exporter opened
2024-11-29T13:37:03.992Z
2024-11-29 13:37:03.963 [Broker-0] [zb-fs-workers-0] [Exporter-1] WARN
2024-11-29T13:37:03.992Z
io.camunda.zeebe.broker.exporter - Failed to update exporter position when skipping filtered record, can be skipped, but may indicate an issue if it occurs often
2024-11-29T13:37:03.992Z
io.camunda.zeebe.db.ZeebeDbException: Unexpected error occurred during RocksDB transaction.
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.db.impl.rocksdb.transaction.DefaultTransactionContext.runInTransaction(DefaultTransactionContext.java:40) ~[zeebe-db-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.ensureInOpenTransaction(TransactionalColumnFamily.java:343) ~[zeebe-db-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.get(TransactionalColumnFamily.java:139) ~[zeebe-db-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.broker.exporter.stream.ExportersState.findExporterStateEntry(ExportersState.java:89) ~[zeebe-broker-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.broker.exporter.stream.ExportersState.setExporterState(ExportersState.java:47) ~[zeebe-broker-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.broker.exporter.stream.ExporterContainer.updateExporterState(ExporterContainer.java:167) ~[zeebe-broker-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.broker.exporter.stream.ExporterContainer.updateExporterState(ExporterContainer.java:155) ~[zeebe-broker-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.broker.exporter.stream.ExporterContainer.updatePositionOnSkipIfUpToDate(ExporterContainer.java:147) ~[zeebe-broker-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.broker.exporter.stream.ExporterContainer.exportRecord(ExporterContainer.java:224) ~[zeebe-broker-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.broker.exporter.stream.RecordExporter.export(RecordExporter.java:78) ~[zeebe-broker-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.scheduler.retry.BackOffRetryStrategy.run(BackOffRetryStrategy.java:51) ~[zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.scheduler.ActorJob.invoke(ActorJob.java:83) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.scheduler.ActorJob.execute(ActorJob.java:42) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.scheduler.ActorTask.execute(ActorTask.java:122) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.scheduler.ActorThread.executeCurrentTask(ActorThread.java:130) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.scheduler.ActorThread.doWork(ActorThread.java:108) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.scheduler.ActorThread.run(ActorThread.java:227) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
Caused by: org.rocksdb.RocksDBException: While open a file for appending: /usr/local/zeebe/data/raft-partition/partitions/1/runtime/000152.sst: No such file or directory
2024-11-29T13:37:03.992Z
at org.rocksdb.Transaction.commit(Native Method) ~[rocksdbjni-9.4.0.jar:?]
2024-11-29T13:37:03.992Z
at org.rocksdb.Transaction.commit(Transaction.java:216) ~[rocksdbjni-9.4.0.jar:?]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.db.impl.rocksdb.transaction.ZeebeTransaction.commitInternal(ZeebeTransaction.java:147) ~[zeebe-db-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.db.impl.rocksdb.transaction.DefaultTransactionContext.runInNewTransaction(DefaultTransactionContext.java:62) ~[zeebe-db-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.db.impl.rocksdb.transaction.DefaultTransactionContext.runInTransaction(DefaultTransactionContext.java:33) ~[zeebe-db-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
… 16 more
2024-11-29T13:37:03.996Z
2024-11-29 13:37:03.963 [Broker-0] [zb-fs-workers-1] [SnapshotStore-1] WARN
2024-11-29T13:37:03.996Z
io.camunda.zeebe.snapshots.impl.FileBasedTransientSnapshot - Unexpected exception on taking snapshot (FileBasedSnapshotId{index=475012, term=2, processedPosition=475012, exporterPosition=475013, brokerId=Optional[0]})
2024-11-29T13:37:03.996Z
io.camunda.zeebe.db.ZeebeDbException: Failed to take snapshot in path /usr/local/zeebe/data/raft-partition/partitions/1/snapshots/475012-2-475012-475013-0.
2024-11-29T13:37:03.996Z
at io.camunda.zeebe.db.impl.rocksdb.transaction.ZeebeTransactionDb.createSnapshot(ZeebeTransactionDb.java:178) ~[zeebe-db-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.996Z
at io.camunda.zeebe.broker.system.partitions.impl.StateControllerImpl.lambda$takeSnapshot$4(StateControllerImpl.java:251) ~[zeebe-broker-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.996Z
at io.camunda.zeebe.snapshots.impl.FileBasedTransientSnapshot.takeInternal(FileBasedTransientSnapshot.java:77) ~[zeebe-snapshots-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.996Z
at io.camunda.zeebe.snapshots.impl.FileBasedTransientSnapshot.lambda$take$0(FileBasedTransientSnapshot.java:62) ~[zeebe-snapshots-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.996Z
at io.camunda.zeebe.scheduler.ActorJob.invoke(ActorJob.java:83) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.996Z
at io.camunda.zeebe.scheduler.ActorJob.execute(ActorJob.java:42) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.996Z
at io.camunda.zeebe.scheduler.ActorTask.execute(ActorTask.java:122) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.996Z
at io.camunda.zeebe.scheduler.ActorThread.executeCurrentTask(ActorThread.java:130) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.996Z
at io.camunda.zeebe.scheduler.ActorThread.doWork(ActorThread.java:108) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.996Z
at io.camunda.zeebe.scheduler.ActorThread.run(ActorThread.java:227) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.996Z
Caused by: org.rocksdb.RocksDBException: While open a file for appending: /usr/local/zeebe/data/raft-partition/partitions/1/runtime/000152.sst: No such file or directory
2024-11-29T13:37:03.996Z
at org.rocksdb.Checkpoint.createCheckpoint(Native Method) ~[rocksdbjni-9.4.0.jar:?]
2024-11-29T13:37:03.996Z
at org.rocksdb.Checkpoint.createCheckpoint(Checkpoint.java:50) ~[rocksdbjni-9.4.0.jar:?]
2024-11-29T13:37:03.996Z
at io.camunda.zeebe.db.impl.rocksdb.transaction.ZeebeTransactionDb.createSnapshot(ZeebeTransactionDb.java:175) ~[zeebe-db-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.996Z
… 9 more
2024-11-29T13:37:03.997Z
2024-11-29 13:37:03.997 [Broker-0] [zb-actors-0] [SnapshotDirector-1] ERROR
2024-11-29T13:37:03.997Z
io.camunda.zeebe.logstreams.snapshot - Failed to take a snapshot for StreamProcessor-1
2024-11-29T13:37:03.997Z
io.camunda.zeebe.db.ZeebeDbException: Failed to take snapshot in path /usr/local/zeebe/data/raft-partition/partitions/1/snapshots/475012-2-475012-475013-0.
2024-11-29T13:37:03.997Z
at io.camunda.zeebe.db.impl.rocksdb.transaction.ZeebeTransactionDb.createSnapshot(ZeebeTransactionDb.java:178) ~[zeebe-db-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.997Z
at io.camunda.zeebe.broker.system.partitions.impl.StateControllerImpl.lambda$takeSnapshot$4(StateControllerImpl.java:251) ~[zeebe-broker-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.997Z
at io.camunda.zeebe.snapshots.impl.FileBasedTransientSnapshot.takeInternal(FileBasedTransientSnapshot.java:77) ~[zeebe-snapshots-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.997Z
at io.camunda.zeebe.snapshots.impl.FileBasedTransientSnapshot.lambda$take$0(FileBasedTransientSnapshot.java:62) ~[zeebe-snapshots-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.997Z
at io.camunda.zeebe.scheduler.ActorJob.invoke(ActorJob.java:83) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.997Z
at io.camunda.zeebe.scheduler.ActorJob.execute(ActorJob.java:42) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.997Z
at io.camunda.zeebe.scheduler.ActorTask.execute(ActorTask.java:122) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.997Z
at io.camunda.zeebe.scheduler.ActorThread.executeCurrentTask(ActorThread.java:130) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.997Z
at io.camunda.zeebe.scheduler.ActorThread.doWork(ActorThread.java:108) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.997Z
at io.camunda.zeebe.scheduler.ActorThread.run(ActorThread.java:227) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.997Z
Caused by: org.rocksdb.RocksDBException: While open a file for appending: /usr/local/zeebe/data/raft-partition/partitions/1/runtime/000152.sst: No such file or directory
2024-11-29T13:37:03.997Z
at org.rocksdb.Checkpoint.createCheckpoint(Native Method) ~[rocksdbjni-9.4.0.jar:?]
2024-11-29T13:37:03.997Z
at org.rocksdb.Checkpoint.createCheckpoint(Checkpoint.java:50) ~[rocksdbjni-9.4.0.jar:?]
2024-11-29T13:37:03.997Z
at io.camunda.zeebe.db.impl.rocksdb.transaction.ZeebeTransactionDb.createSnapshot(ZeebeTransactionDb.java:175) ~[zeebe-db-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.997Z
… 9 more

Would you please share all the links you have used and configuration you made for Hazelcast Exporter.

Hi Sir,

Here’s my Zeebe environment configuration. I didn’t set up anything related to Hazelcast explicitly, so I believe it’s just using the default settings:

ZEEBE_BROKER_DATA_DISKUSAGECOMMANDWATERMARK: 0.998
ZEEBE_BROKER_DATA_DISKUSAGEREPLICATIONWATERMARK: 0.999
ZEEBE_BROKER_EXPORTERS_HAZELCAST_JARPATH: /exporter/zeebe-hazelcast-exporter.jar
ZEEBE_BROKER_EXPORTERS_HAZELCAST_CLASSNAME: io.zeebe.hazelcast.exporter.HazelcastExporter
ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_CLASSNAME: io.camunda.zeebe.exporter.ElasticsearchExporter
ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_URL: http://elasticsearch:9200
ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_BULK_SIZE: 1

thank you.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.