I am a new user of Camunda. After deploying Zeebe to AWS using Fargate and installing the Hazelcast exporter to use the Simple Zeebe Monitor, I mounted EFS to the zeebe_data folder. However, I am encountering an error, and no job worker can connect to Zeebe.
Could anyone please help me understand why this is happening and how to prevent the error from occurring again?
Thank you for your assistance.
2024-11-29T13:36:09.672Z
io.camunda.zeebe.broker.exporter.elasticsearch - Exporter opened
2024-11-29T13:37:03.992Z
2024-11-29 13:37:03.963 [Broker-0] [zb-fs-workers-0] [Exporter-1] WARN
2024-11-29T13:37:03.992Z
io.camunda.zeebe.broker.exporter - Failed to update exporter position when skipping filtered record, can be skipped, but may indicate an issue if it occurs often
2024-11-29T13:37:03.992Z
io.camunda.zeebe.db.ZeebeDbException: Unexpected error occurred during RocksDB transaction.
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.db.impl.rocksdb.transaction.DefaultTransactionContext.runInTransaction(DefaultTransactionContext.java:40) ~[zeebe-db-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.ensureInOpenTransaction(TransactionalColumnFamily.java:343) ~[zeebe-db-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.get(TransactionalColumnFamily.java:139) ~[zeebe-db-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.broker.exporter.stream.ExportersState.findExporterStateEntry(ExportersState.java:89) ~[zeebe-broker-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.broker.exporter.stream.ExportersState.setExporterState(ExportersState.java:47) ~[zeebe-broker-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.broker.exporter.stream.ExporterContainer.updateExporterState(ExporterContainer.java:167) ~[zeebe-broker-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.broker.exporter.stream.ExporterContainer.updateExporterState(ExporterContainer.java:155) ~[zeebe-broker-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.broker.exporter.stream.ExporterContainer.updatePositionOnSkipIfUpToDate(ExporterContainer.java:147) ~[zeebe-broker-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.broker.exporter.stream.ExporterContainer.exportRecord(ExporterContainer.java:224) ~[zeebe-broker-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.broker.exporter.stream.RecordExporter.export(RecordExporter.java:78) ~[zeebe-broker-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.scheduler.retry.BackOffRetryStrategy.run(BackOffRetryStrategy.java:51) ~[zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.scheduler.ActorJob.invoke(ActorJob.java:83) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.scheduler.ActorJob.execute(ActorJob.java:42) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.scheduler.ActorTask.execute(ActorTask.java:122) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.scheduler.ActorThread.executeCurrentTask(ActorThread.java:130) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.scheduler.ActorThread.doWork(ActorThread.java:108) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.scheduler.ActorThread.run(ActorThread.java:227) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
Caused by: org.rocksdb.RocksDBException: While open a file for appending: /usr/local/zeebe/data/raft-partition/partitions/1/runtime/000152.sst: No such file or directory
2024-11-29T13:37:03.992Z
at org.rocksdb.Transaction.commit(Native Method) ~[rocksdbjni-9.4.0.jar:?]
2024-11-29T13:37:03.992Z
at org.rocksdb.Transaction.commit(Transaction.java:216) ~[rocksdbjni-9.4.0.jar:?]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.db.impl.rocksdb.transaction.ZeebeTransaction.commitInternal(ZeebeTransaction.java:147) ~[zeebe-db-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.db.impl.rocksdb.transaction.DefaultTransactionContext.runInNewTransaction(DefaultTransactionContext.java:62) ~[zeebe-db-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
at io.camunda.zeebe.db.impl.rocksdb.transaction.DefaultTransactionContext.runInTransaction(DefaultTransactionContext.java:33) ~[zeebe-db-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.992Z
… 16 more
2024-11-29T13:37:03.996Z
2024-11-29 13:37:03.963 [Broker-0] [zb-fs-workers-1] [SnapshotStore-1] WARN
2024-11-29T13:37:03.996Z
io.camunda.zeebe.snapshots.impl.FileBasedTransientSnapshot - Unexpected exception on taking snapshot (FileBasedSnapshotId{index=475012, term=2, processedPosition=475012, exporterPosition=475013, brokerId=Optional[0]})
2024-11-29T13:37:03.996Z
io.camunda.zeebe.db.ZeebeDbException: Failed to take snapshot in path /usr/local/zeebe/data/raft-partition/partitions/1/snapshots/475012-2-475012-475013-0.
2024-11-29T13:37:03.996Z
at io.camunda.zeebe.db.impl.rocksdb.transaction.ZeebeTransactionDb.createSnapshot(ZeebeTransactionDb.java:178) ~[zeebe-db-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.996Z
at io.camunda.zeebe.broker.system.partitions.impl.StateControllerImpl.lambda$takeSnapshot$4(StateControllerImpl.java:251) ~[zeebe-broker-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.996Z
at io.camunda.zeebe.snapshots.impl.FileBasedTransientSnapshot.takeInternal(FileBasedTransientSnapshot.java:77) ~[zeebe-snapshots-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.996Z
at io.camunda.zeebe.snapshots.impl.FileBasedTransientSnapshot.lambda$take$0(FileBasedTransientSnapshot.java:62) ~[zeebe-snapshots-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.996Z
at io.camunda.zeebe.scheduler.ActorJob.invoke(ActorJob.java:83) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.996Z
at io.camunda.zeebe.scheduler.ActorJob.execute(ActorJob.java:42) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.996Z
at io.camunda.zeebe.scheduler.ActorTask.execute(ActorTask.java:122) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.996Z
at io.camunda.zeebe.scheduler.ActorThread.executeCurrentTask(ActorThread.java:130) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.996Z
at io.camunda.zeebe.scheduler.ActorThread.doWork(ActorThread.java:108) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.996Z
at io.camunda.zeebe.scheduler.ActorThread.run(ActorThread.java:227) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.996Z
Caused by: org.rocksdb.RocksDBException: While open a file for appending: /usr/local/zeebe/data/raft-partition/partitions/1/runtime/000152.sst: No such file or directory
2024-11-29T13:37:03.996Z
at org.rocksdb.Checkpoint.createCheckpoint(Native Method) ~[rocksdbjni-9.4.0.jar:?]
2024-11-29T13:37:03.996Z
at org.rocksdb.Checkpoint.createCheckpoint(Checkpoint.java:50) ~[rocksdbjni-9.4.0.jar:?]
2024-11-29T13:37:03.996Z
at io.camunda.zeebe.db.impl.rocksdb.transaction.ZeebeTransactionDb.createSnapshot(ZeebeTransactionDb.java:175) ~[zeebe-db-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.996Z
… 9 more
2024-11-29T13:37:03.997Z
2024-11-29 13:37:03.997 [Broker-0] [zb-actors-0] [SnapshotDirector-1] ERROR
2024-11-29T13:37:03.997Z
io.camunda.zeebe.logstreams.snapshot - Failed to take a snapshot for StreamProcessor-1
2024-11-29T13:37:03.997Z
io.camunda.zeebe.db.ZeebeDbException: Failed to take snapshot in path /usr/local/zeebe/data/raft-partition/partitions/1/snapshots/475012-2-475012-475013-0.
2024-11-29T13:37:03.997Z
at io.camunda.zeebe.db.impl.rocksdb.transaction.ZeebeTransactionDb.createSnapshot(ZeebeTransactionDb.java:178) ~[zeebe-db-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.997Z
at io.camunda.zeebe.broker.system.partitions.impl.StateControllerImpl.lambda$takeSnapshot$4(StateControllerImpl.java:251) ~[zeebe-broker-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.997Z
at io.camunda.zeebe.snapshots.impl.FileBasedTransientSnapshot.takeInternal(FileBasedTransientSnapshot.java:77) ~[zeebe-snapshots-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.997Z
at io.camunda.zeebe.snapshots.impl.FileBasedTransientSnapshot.lambda$take$0(FileBasedTransientSnapshot.java:62) ~[zeebe-snapshots-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.997Z
at io.camunda.zeebe.scheduler.ActorJob.invoke(ActorJob.java:83) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.997Z
at io.camunda.zeebe.scheduler.ActorJob.execute(ActorJob.java:42) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.997Z
at io.camunda.zeebe.scheduler.ActorTask.execute(ActorTask.java:122) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.997Z
at io.camunda.zeebe.scheduler.ActorThread.executeCurrentTask(ActorThread.java:130) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.997Z
at io.camunda.zeebe.scheduler.ActorThread.doWork(ActorThread.java:108) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.997Z
at io.camunda.zeebe.scheduler.ActorThread.run(ActorThread.java:227) [zeebe-scheduler-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.997Z
Caused by: org.rocksdb.RocksDBException: While open a file for appending: /usr/local/zeebe/data/raft-partition/partitions/1/runtime/000152.sst: No such file or directory
2024-11-29T13:37:03.997Z
at org.rocksdb.Checkpoint.createCheckpoint(Native Method) ~[rocksdbjni-9.4.0.jar:?]
2024-11-29T13:37:03.997Z
at org.rocksdb.Checkpoint.createCheckpoint(Checkpoint.java:50) ~[rocksdbjni-9.4.0.jar:?]
2024-11-29T13:37:03.997Z
at io.camunda.zeebe.db.impl.rocksdb.transaction.ZeebeTransactionDb.createSnapshot(ZeebeTransactionDb.java:175) ~[zeebe-db-8.6.5.jar:8.6.5]
2024-11-29T13:37:03.997Z
… 9 more