Why do I want to use exporters?
Camunda 8 is based on a highly loose-coupled architecture - and I have On-Premise projects where I’m interested in fine-grained events from the engine like Incidents, Human-Task Jobs, etc. and where I do not want to poll any API but rather get notified immediately. To summarize: workers and connectors are great, but there are times where I want more.
Why Redis?
Redis (https://redis.io/) offers a lot of options to us. I can use it as fast in-memory technology and simple transport layer. I can optionally add persistence, but it is not required. I have Redis streams at my hand which are great when I want to use multiple consumers and scaling on the event receiver side. In Java the Lettuce client (https://lettuce.io/) offers great connectivity features out of the box, so that the exporter itself is easy to maintain. …
If neither the Hazelcast nor the Kafka exporter fits your exact needs, this could be the one you’re looking for.
Please be aware: the project is still in incubation phase and might change in some parts. If you like the project, give it a star on GitHub. If you have any suggestions, improvements, etc: there are GitHub issues and GitHub pull requests at your hand. If you have something much better to connect Zeebe to Redis, go for it and let us know. Happy to get you involved and to hear from you.
@VonDerBeck Thanks. This is cool! I am dreaming of a “minimal” Zeebe-setup for mostly headless automation setups, and this would be perfect.
I tried this out with Python consumer, populating a SQLite database, and it was fun.
What kind of Redis setup are you using in production? Have you compared AOF to RDB for persistence? Have you tried to let consumer to do XTRIM once it has consumed the data? And any thoughts about TCP port vs Unix sockets with Redis server?
Some of your questions are quick to answer (at least for me):
Production Setup: it will be some time before we are in production. We are dealing with something bigger. Persistence will likely be a topic. Considering AOF / RDB - not yet done. So if you want to share some experience - you’re welcome.
TCP vs Unix sockets - not a topic for us. We’re running in a distributed cloud environment. TCP will be the only option.
The question currently on my mind is the question of the most efficient way to keep the streams as small as possible and to delete processed data in a timely manner. Always keeping in mind that we have scaled consumers with consumer groups and we only want to really delete data once it has been consumed by all different participants.
The current quick win is an option within the Java Connector being able to optionally call XDEL after a specific message has been acknowledged. Which is not yet what I really want - it does not consider multiple consumer groups and in terms of performance I do not yet know if this is really the way to go.
The exporter itself currently is able to XTRIM entries based on time (thus deleting too old entries). As stated in the project documentation we’re not yet done here.
But there might be a way to possibly achieve the real thing…
We can use XINFO GROUPS to get all consumer groups, including the “last-delivered-id”, the minimum of these tells us up to where we can delete. Hence we need to run XTRIM with MINID on a regular base using the XINFO GROUPS result as base. I still have to try that out.
What do you think? Am I missing something? The topic of “delete after acknowledge” is widely discussed in all sorts of Redis related threads - in very differenty flavours.
Personally I’d like try AOF, because I have 20 years of good experiences on AOF object database of particular brand, and therefore it sounds like an approach I could trust. That said, I have no experience about it with Redis, and I wonder how it manages to rewrite the files under high load. (According to docs, probably fine after version 7.0.0 with splitting it to multiple files.)
That sounds like something that you could eventually support directly in the plugin? That sounds superior to the current XTRIM based on time. But how to initialize the groups to wait for? Should that be in the plugin configuration or preconfigured to Redis instance before starting Zeebe?
@datakurre:
That sounds like something that you could eventually support directly in the plugin? That sounds superior to the current XTRIM based on time. But how to initialize the groups to wait for? Should that be in the plugin configuration or preconfigured to Redis instance before starting Zeebe?
XINFO GROUPS returns a list of all consumer groups having consumed data together with their “last-delivered-id”. So it doesn’t seem to be a good idea to delete data where absolutely no consumer group at all has consumed any data. In this case I could optionally fall back to time based cleanup configured with a bigger time to live value. That’s easy. As far as this is an optional feature and everone knows what to expect it could be a way to go.
An edge case is, where I have an unknown consumer group which has never connected to Redis but where I want to startup this consumer at some time in the future and want to receive historic entries. Or where I had a consumer once connected but now being taken down forever.
In order to manage that I would need a complete list of potential consumer groups to consider during cleanup. The downside of such a consumer list parameter is indeed that I need to configure this stuff before starting Zeebe. And the configuration needs to change everytime I want to add or remove a new consumer group. Not sure if I like this. Rather not…!
A good compromise could be to combine the simple “delete-after-acknowledge” algorithm like described above with some time to live values, e.g. minTimeToLive and maxTimeToLive. This should leave enough room for all sorts of scenarios.
With Zeebe, it is a tempting idea to store the complete event history, to be able to deploy new applications with their own “replayed” databases later. But most probably Redis is not the right tool for that. (I know that Kafka is, and personally I’d like to try out RabbitMQ Streams for that in the future.)
Also, I don’t have experience with Redis AOF yet with a lot of data, but if it really has to replay the whole AOF when it starts, maxTimeToLive really sounds mandatory to keep AOF size reasonable. (Of course, AOF also needs BGREWRITEAOF to actually rewrite new version of log without expired data.)
Thanks. Exactly, Redis is not the right tool for that. It’s intended as high performance cache distributing events to other parties. Of course this needs some flexibility - but its not intended to replace e.g. Kafka. For other scenarios other exporters are available. Personally I’d like to keep the footprint as small as possible.
