Camunda 7 - multi tenancy with 1 process engine / tenant

Camunda 7 Topics
,
1

Hi:

I see in camunda 7 docs, that multi tenancy with 1 process engine / tenant is a valid configuration. Multi-Tenancy | docs.camunda.org says how to configure for large number of process engines.

Based on some initial tests in a springboot application, for a 2gig memory I am not able to go higher than 30-40 engines.

Camunda Team,
What is number of engines I could scale to per JVM ?
Do you have some ball park numbers for perfomance?

1 Like
2

Do you want to have one or many engines? I can’t understand your question.

3

Hi fml2,

I am using option of 1 engine/DB per tenant in spring boot camunda 7 app. It seems to use a large amount of memory per engine. for 50 engines, i.e. 50 customers, i am seeing about 1gig of heap memory needed.

The link above I shared from camunda 7 docs →

The process engines can be configured in a configuration file or via Java API.
rajans> I am doing config via Java API

Job Executor for Multiple Process Engines

For background execution of processes and tasks, the process engine has a component called job executor. The job executor periodically acquires jobs from the database and submits them to a thread pool for execution. For all process applications on one server, one thread pool is used for job execution. Furthermore, it is possible to share the acquisition thread between multiple engines. This way, resources are still manageable even when a large amount of process engines are used. See the section The Job Executor and Multiple Process Engines for details.
My Question on → large number of process engines are used.
How many process engines can be configured for 2gig of memory for this option I am using - 1 process engine / tenant ??

My observation
I am starting 50 process engines by an addTenant API in a JVM.
I am kicking off 1 workflow in each engine.
I am observing committed heap space is 1.1gigs.
Also the heap number is fluctuating over time. I goes to 1.1gig and reduces to 400megabytes. It again increases to 1.1gigs and again reduces to 400gigs, per GC cycle.

Is there any kind of caching or some engine configuration causing this large memory consumption ?
Has Camunda performance team have any numbers on how many engines can be supported when 1 engine per tenant mode is used?

thanks
rajans

rajans

4

OK, now it’s clear what you mean. I’m sorry, but I’ve never used many engines per JVM. I don’t have any statistics, but my feeling is that it’s an officially possible but a rather seldomly used setup. I’ve only seen one application (and engine) per JVM (but many applications per DB).

JVMs are memory hungry in general; I can’t say how much the camunda engine adds to that (via its inefficient management of something or memory leaks or …).

5

hi:

I am confused by this reply.
Multi-tenancy with 1 engine/tenant is advertised mechanism in Camunda 7 docs.
Are u saying this is NOT supported by Camunda?

Sorry fml2,
I am not sure how these forums work,
Is there a way to get response from official Camunda support team? Or they dont monitor these forums?

rajans

6

This isn’t an official Camunda Support channel - you are likely to only get feedback from other users (I’m also only a user). Some Camunda employees watch and respond to the forums on an “as possible” basis.

So, no there is no way to definitively get a response from Camunda on these forums. If you get a support contract with Camunda, my understanding is that there is a distinct support desk system that you can submit tickets into to get a definitive response from Camunda.

7

What do you mean by that? Can you show some code?

But, as I wrote, I have not seen a setup where multiple process engines were executed in one JVM.

8

You should share few details

  1. What is your use case to run so many process engines in one JVM? Process engine being multithreaded may consume good amount of CPU and RAM. I have seen people using one process engine and running successfully hundreds/thousands of processes in it with proper tuning/configuration. Here is a very good official guide on performance tuning. There is no silver bullet as there are many factors Performance tuning Camunda 7 | Camunda 8 Docs
  2. A sample code will be good.
9

Hi ad_sahota:

  1. We are using camunda 7.21 in an embedded spring boot application.
  2. We have a mandatory requirement for a multi tenant service, to host camunda DB in a separate schema and NOT use the single engine with tenant identifiers.
    So we have chosne one engine per tenant model as described in the multi tenancy docs of 7.x.
  3. So I have programatically created engines on the fly during initial one-time addTenant API.
  4. When workflows are started, I switch to correct engine context as the API or delegate executions carry the tenantId.
  5. What I am observing for the springboot application is for 30 tenants, 1 gig of heap is consumed. Even though load per tenant is small. I need to be able to support upto say 500-1000 customers per deployment.
10

addTenant() API code snippet →

{
TenantEntity tenant = tenantRepository.getTenant(“customer1”);
// following creates the MYSQL schema customer1 and creates all camunda tables
tenantRepoService.initTenantDatabase(tenant);

// get an instance of jobexecutor
SharedJobExecutor sje =jobExecutorManager.getJobExecutorElseDefault(id);

// create and register process engine
ProcessEngine pe = createProcessEngine (tenant, sje);
engineMetricsReporter.initializeEngineMetrics(pe.getName(), pe);
}

// create process engine method
ProcessEngine createProcessEngine(TenantEntity entity, sje) {
SpringProcessEngineConfiguration config =
engineConfiguration.buildConfiguration(plugins, entity … );

ProcessEngine pe = config.buildProcessEngine();
RuntimeContainerDelegate.INSTANCE.get().registerProcessEngine(pe);
engineConfiguration.createTenantAndUserAndAuthz(entity.getId(), entity.getTenantName(), pe);

return pe;

}

// custom shared job executor
public class SharedJobExecutor extends ThreadPoolJobExecutor {
…
}

11

Oh, you create engines and tenats dynamically via API? This is a highly unusual setup IMO (which doesn’t mean it should not work). Sorry, I can’t tell anything about how it should work. I’ve never used that.

I wonder what is your business case that yields such solution.

12

Here are issues I see (In my personal opinion) with your setup:

  1. JVM will be single point of failure for all engines together in one instance of your solution. If JVM crashes due to some reason, all 40 engines will crash.
  2. On very heavy load, the JVM heap can grow very high. I have seen camunda JVM grow upto 6 GB on very high load for a single engine. It depends on your load as well. Imaging you decise to run 1000s of process instance in each engine . You may have to deploy a big machine.

Since you are using separate DB schema for each engine then why not deploy separate Camunda instance with unique engine for each database. This way you can scale individual camunda instance separately as they are decoupled from each other.
If you are using infrstructure as code (docker/kubernetes) then it willl be very easy to manage as well. You can deploy each engine in separate kubernetes namespace. Based on need you can increase the replicas.
You just have to manage separate configs per deployment.

13

@ad_sahota is correct. Camunda will probably tell you that their officially supported path in Spring boot is a single engine per app instance.

We did indeed set up single instance per customer in Kubernetes cluster. We fronted them with nginx proxy to direct traffic correctly.

14

Hi ad_sahota:
Thanks for feedback.

What are your thoughts here …
It seems the 7.22 documentation is explaining a cluster setup here and how to scale →

Do you see how a single Job Executor Thread pool in the diagram is shared across process engines? Does that not imply in a single JVM there are many process engines?

rajans

15

I have not used this setup as the documentation seems like focussed on Tomcat/JakartaEE application servers so cannot comment.
With springboot based setup we prefer single engine per pod for simplicity. We horizontally scale pods based on need.

16

Same here.