If there are multiple jobs in the same partition and some workers are unavailable to serve them, how does Camunda store or queue these jobs for the future? Additionally, how does it retrieve these jobs in the order they arrived in the partition log?
Consider a simple workflow of Start Node, 4 parallel tasks, Join node, End Node.
Can you explain this aspect of your question bit more
Are you talking about a worker going down or unable to access the jobs for some reason?
Mostly Worker going down. Other scenarios can be considered as well of worker not available to serve the job due to some xyz reason.
Active jobs are kept in perpetuity on the broker that owns the partition - specifically in RocksDB.
If there’s no worker to pick up the job, the job just waits.
Got it. How does it save these jobs in the RocksDB and retrieves from it in the sequence that it was arrived? I read that rocksDB stores the data in sorted order and each partition has its own rocks db instance. Plus internally when I saw it is storing activable jobs in a columnFamily.
Please share some light on the internals of how does it do it? Or if you can share any blogs / videos where I can get these internals in detail.
There’s an append only event log for each partition so the order that everything happens is locked into that.
Can you share a few references where I can read about how camunda 8 uses rocks db to do state management?
Sure, it can be found in this section of the docs.