Microservices with Camunda! Multi-Engine Data Service - Looking for some testers

Hello everyone!

As I have been hearing more and more chatter recently about the use of Camunda in microservice models, I wanted to share some tooling that we have been working on that was built for using many (dozens, hundreds, thousands +++) of Camunda instances/engines and clusters of engines.

It could be used for a single engine, a cluster, or many different microservices that are built with a camunda engine.

We have been calling it the “Multi-Engine Data Service”, where it provides a custom history back end for the engine and is fully integrated into the camunda transaction system.

some of the features:

  1. Works with Camunda Run, Spring Boot, and other core camunda distributions.

  2. For developers building custom APIs, it provides modernized “DTOs” for all history events.

  3. In additional to all history events, Deployments, Deployment Resources, Process Definitions, Decision Decisions, and Decision Requirement Definitions are all sent to the service. (Process Definitions are additionally parsed into JSON, so you can natively query for history data based on BPMN configurations in the XML: Example: “Find all activity instances where the activity instance has a Start Event Listener that executes Javascript and contains […this…] snippet of code”).

  4. All history gets tagged with the engine, cluster, transaction id, etc. The purpose of the design is to support distributed history that can be collected into a single queryable source (including realtime)

  5. The history data supports complex join scenarios and nested data structures with all history events. So you can easily ask for data such as: "Give me all process instances that have JSON variables with the property price between 1$ and 100$, and return process variables, activity instances, and incidents for each process instance).

  6. JSON variables and other complex Variables are stored in queryable structures rather than camunda’s byte arrays: therefore you can query Spin/JSON variables in their JSON formats.

  7. Robust security filtering is provided as part of the History Service, so you can create complex security policy rules for data access that were not previously possible in Camunda’s History Services. (nested AND and OR queries, Identity Links logic, JSON variable data, Other Java Object data (such as custom business objects used during a process instance, etc). All of this allows you to create security filtering rules that is executed at the DB level and is agnostic to Camunda’s users, roles, groups, etc.

  8. The history data gets stored in MongoDB so you can take advantage of all of the Aggregation Framework capabilities and Change Streams.

  9. You can connect your tools such as Grafana, Tableau, PowerBi, etc directly to the data and query in realtime.

  10. Highly flexible Data Importer used transfer of data from existing Camunda instances into the data service. This allows you to create highly customized import queries from your existing camunda engines and import into the new service.

The overall goal of the build was to create the following capabilities:

  1. Disconnect from the SQL structures enforced by the engine
  2. Enable complex business queries in native formats that all support realtime querying
  3. Provide a camunda-agnostic data query filtering and optional security layers that can enforce the very common complex security rules that the built-in Camunda Authorization system cannot support.
  4. Bring together data from any engine regardless of location and use.
  5. Support non-persistent engine scenarios such as single execution engine life cycles (CLI executions, etc).
  6. Connect existing organizational report tooling to these realtime datasources: Grafana, Tableau, PowerBi, Metabase.

So we are looking for some testers from groups that have larger sets of deployments. Looking to expand the usage and grow its abilities.

If you are interested please let me know!

6 Likes

This sounds really fantastic, thanks for sharing I can think of a bunch of people who I could pass this on to and I’ll do that now :slight_smile:

Hi Stephe. Wow - that sounds pretty interesting! It might be a great sample implementation of “Custom Centralized Monitoring” I describe as one option for E2E visibility in my upcoming book: https://learning.oreilly.com/library/view/practical-process-automation/9781492061441/.

Do I assume correctly that you focused on getting visibility into many engines, not so much the E2E process spanning multiple engines?

Out of curiosity: What’s the status of this exactly? E.g. is this a community extension on GitHub maintained by you, everybody interested can try out? Or is this some closed source code you have from another project and want to setup kind of a POC scenario with another potential user? Or…?

Thanks! Bernd

@BerndRuecker from a “monitoring” perspective you could use it easily for spanning multiple engines as all of the data can be merged into single query results: “Find all process instances/activitity instances/variables, etc from process definitions/instances with specific configuration, variable values, activity interactions, etc” (so for example an Order ID).
Because it supports multi-engine you only need to worry about “E2E processes that span multiple engines” as a function of a business data query (DB query).

I am working to get it opened publicly or what is looking more like partially opened into GitHub. There is lots of closed source I have been working to strip out ;).

It currently works as a drop-in/configurable process engine plugin that runs as a additional or replacement History Event Handler. If there is enough interest we can see about shortening timelines.

The query services can be run within an engine or within other apps that connect to the same data source (as per the diagram).

1 Like

Thanks for the clarification - sounds really impressive. I could imagine that we get a hen and egg problem - as it is hard to generate enough interest without more visibility into what it is :wink: Maybe you could start with a simple readme that explains the user side of things to have something to link to?

1 Like

My organization is just getting started with Camunda, but this sounds very interesting and I’d like to be kept up to date with what’s going on.

Looking forward to reading a copy!

Hey

Finally circling back around on this:

You can check out the latest. Will update with docs and code updates as I can release them.