Fail on startup if database is not upgraded

Camunda 7 Topics Discussion & Questions
1

Environment

Spring Boot application with Camunda Process Engine embedded (Spring Boot 2.6.4, Camunda Process Engine 7.16.0).

Problem Statement

Some time ago we upgraded Camunda from version 7.14.0 to 7.16.0.
After a while we faced an issue on our QA servers while trying to resolve a business incident:

Cause: java.sql.BatchUpdateException: Batch entry 0 insert into ACT_RU_TASK_METER_LOG (ID_, TIMESTAMP_, ASSIGNEE_HASH_)
    values (
    '...'
    ) was aborted: ERROR: relation "act_ru_task_meter_log" does not exist

The issue was caused by the not upgraded database.
We were confused by the property camunda.bpm.database.schema-update (forum discussion - DB schema breaking change when upgrade from 7.13 to 7.15).

Luckily we didn’t face the problem in production, otherwise, it could lead to very serious problems.

Looks like we were not the only ones who were confused by the upgrade mechanism - forum topics (don’t have the ability to put more links as the new user): Camunda 7.6.0 failed on database created by 7.5.0, Issues with upgrading Camunda BPM from 7.6 to 7.7, etc.

In our opinion, the most significant flaw is not the upgrade procedure by itself, but rather the lack of database version validation on service startup.

We were surprised that there is no such validation - usually, it makes sense to verify that the service is compatible with the underlying database and to immediately fail to avoid unpredictable results up to data corruption.

Probably there are some database version compatibility checks but they are disabled by default?
If not, are they placed somewhere in the roadmap maybe?

Best Regards,
Timur

2

Hi @tnizametdinov

Thanks a lot for posting this topic, i completely agree with you, i think it would be something really useful and long overdue.

I’ll look into if there’s already a ticket for this feature and let you know.

1 Like
3

Firstly, Thanks for referring to my topic :grin:

TL,DR
From what I understand, till now full auto validation from the application itself is not supported (read bellow why).
But starting from 7.16 you can have semi-auto installation, update and validation of your schema using Liquibase.

As I faced a similar issue as mentioned in the referred link, I asked the same question if camunda has such a validation tool like full integration with Liquibase.

I think starting from 7.16, Camunda engine starting the integration with Liquibase.

In breif, Liquibase it’s DB migration tool that keeps track of database schema changes and auto update the targeted DB to be compatible with the new releases.

Liquibase makes sure that your current scheme is full compatible with the new release automatically from the application itself as one of the lequibase features.
But Camunda teams believe that auto updating the scheme by the application is something you should not do especially in production as the least privilege principal states so.

So using Liquibase with camunda doesn’t give us the full automation capability we want but at least we can make sure that our schema is updated and compatible by running only one Liquibase command.

Tip: In order to not have human intervention, you can integrate Liquibase in the deployment build pipeline as well!

Read more on how to start using the feature from the link below.

Regards,

1 Like
4

You are welcome @moath.alshorman :wink:

Thanks for the explanation.
In other services, we are using liquibase “competitor” - flyway.
The schema update mechanism is fully automated - the migration is started from the side container as a part of a regular deployment described as a helm chart.

Still, in addition to the automated updates, we have a special code that checks that the actual and expected database versions match and fail if not on the application start.

Basically, would be nice to have something similar in Camunda.

5