Faster startup time

Camunda 7 Topics Discussion & Questions
1

I’ve been investigating the startup time of a bare-bones Camunda Platform process engine (no job executor, history cleanup etc) to try and reduce it. On my box it takes a little over 5 seconds with a local Postgres backend. I’ve tried some things already, such as removing the BootstrapEngineCommand and blocking telemetry, but the logging still shows a 3 second period where nothing happens. Can anyone enlighten me what happens in that 3 second period?

// logging
2021-04-01 11:34:28.949 {} DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Fill pool skipped, pool is at sufficient level.
2021-04-01 11:34:29.149 {} INFO  org.camunda.feel.FeelEngine - Engine created. [value-mapper: CompositeValueMapper(List(org.camunda.feel.impl.JavaValueMapper@7d2a6eac)), function-provider: org.camunda.bpm.dmn.feel.impl.scala.function.CustomFunctionTransformer@3003697, clock: SystemClock, configuration: Configuration(false)]
2021-04-01 11:34:29.266 {} DEBUG org.camunda.bpm.engine.cfg - ENGINE-12004 Database product name PostgreSQL
2021-04-01 11:34:29.266 {} DEBUG org.camunda.bpm.engine.cfg - ENGINE-12005 Database type postgres
2021-04-01 11:34:29.854 {} DEBUG org.apache.ibatis.logging.LogFactory - Logging initialized using 'class org.apache.ibatis.logging.slf4j.Slf4jImpl' adapter.
2021-04-01 11:34:32.679 {} DEBUG org.camunda.bpm.engine.cmd - ENGINE-13005 Starting command -------------------- SchemaOperationsProcessEngineBuild ----------------------
2021-04-01 11:34:32.682 {} DEBUG org.camunda.bpm.engine.cmd - ENGINE-13009 opening new command context
2021-04-01 11:34:32.699 {} DEBUG org.apache.ibatis.transaction.jdbc.JdbcTransaction - Opening JDBC Connection    ....

// code
HikariConfig config = new HikariConfig();
config.setJdbcUrl("jdbc:postgresql://localhost:5432/camunda");
config.setUsername("postgres");
config.setPassword("<pw>");
config.setMinimumIdle(1);
config.setMaximumPoolSize(10);
HikariDataSource ds = new HikariDataSource(config);
// various stuff to remove telemetry/history
ProcessEngineBootstrapCommand bootstrapEmpty = new ProcessEngineBootstrapCommand() {
  @Override
  public Void execute(CommandContext commandContext) {
    return null;
  }
};
StandaloneProcessEngineConfiguration processEngine = new StandaloneProcessEngineConfiguration() {
  @Override
  protected void initTelemetry() {
  }

  @Override
  protected void initTelemetryData() {
  }
};
processEngine.setDataSource(ds);
processEngine.setDatabaseSchemaUpdate(ProcessEngineConfiguration.DB_SCHEMA_UPDATE_FALSE);
processEngine.setJobExecutorActivate(false);
processEngine.setProcessEngineBootstrapCommand(bootstrapEmpty);
processEngine.setHistoryCleanupEnabled(false);
processEngine.setProcessEngineBootstrapCommand(bootstrapEmpty);
processEngine.buildProcessEngine();

As can be seen in the logging, there is a three second gap before ---SchemaOperationsProcessEngineBuild--- is printed. What happens at that point?

2

I think this may be due to parsing MyBatis mapping data. It does seem very slow though. Any thoughts?

3

Hi! I am interested in this topic as well. Did you get any further insights yet?

4

Hi @boblondon,

recently I tried out a java profiler and I can confirm your thoughts. Most CPU time during startup is spend on the iBatis mapping.

Hope this helps, Ingo

5

Hi @Ingo_Richtsmeier , Hi @boblondon ,

I also did a performance analysis some time ago related to our Micronaut Camunda Integration and came to the same conclusion. The problem is the sequential parsing of the MyBatis mapping files (over 50!).

I also hacked MyBatis to parse them in parallel and the time dropped down quite a bit. However, I didn’t follow this to the end. Maybe I should have a look again…

See also Performance-Analysis of Camunda Process Engine · Issue #19 · NovatecConsulting/micronaut-camunda-bpm · GitHub

Kind Regards
Tobias

1 Like
6

Out of interest: Why is it important that the application starts in 5 seconds (and not in e.g. 15)? Process applications usually run for a long time, hence in my view it should be neglectable how long the start up takes (as long as it’s reasonably short).

This is not to say that one should not reduce it if it’s obvious how to do this, but in general I’d not put much effort into it.

7

@fml2 , if you only deploy once in a while it will probably be okay for you.

However, if you embed Camunda into your microservice and you start it locally quite often during development fast startup times becomes getting more important. And if your test suite requires the application context to be built multiple times (not optimal but often the case) then it starts getting longer and longer locally and also in the continuous integration (CI) pipeline.

If a Micronaut application usually starts up in less than a second then the bootstrapping of the process engine just feels slow. I’d prefer to get rid of that bottleneck :wink:

8

Hi @fml2,

if you get faster feedback from your JUnit tests, you will be even more productive…

While testing processes you will start the engine very often.

Hope this helps, Ingo

1 Like
9

Thanks for the confirmation of it mostly being MyBatis. This agrees with my work.

We intend to run Camunda on AWS Lambda where startup times really matter. We found the setUseSharedSqlSessionFactory setting on ProcessEngineConfiguration which helps a lot as subsequent engine startups take 0.1 to 0.2 seconds. However, in order for this to work there has to be a single DataSource shared between all the process engines. This was tricky for us as we have a multi-tenant setup where every call to AWS Lambda may be a different tenant. Our solution is a multi-tenant DataSource similar to one in Spring. Thus MyBatis sees a shared DataSource but the actual underlying DataSource supplying connections is changed each request.

@tobiasschaefer I like what you’ve with the mappings there in Micronaut. I’ll have to see if it works for us, or if we can push parallel parsing into MyBatis itself.

10