Operate 1.2.5 complains it can't upgrade from Zeebe 1.0

I’m trying to evaluate Operate on our self-hosted Zeebe Cluster in AWS.

I was able to start it, but then I got this message

SchemaStartup: migrate schema.
[...]
ImportJob                        : Import is not possible for Zeebe version: 1.0
io.camunda.operate.exceptions.OperateRuntimeException: Import is not possible for Zeebe version: 1.0

The UI is empty, even if I see 100k+ documents in elasticsearch.

Background: initially, I was on Zeebe 1.0.5 … I did upgrade straight to 1.2.5, without any noticeable issues. After that, I’m now trying to run Operate but can’t get beyond this error message.

What can I do?


[update]
I realized in older log statements, that indeed, there are no update scripts for 1.0 versions, just 1.1 ones.
Unfortunately, when I switch back to 1.1.6 now, I got other issues

Error creating bean with name 'schemaStartup': 
Invocation of init method failed; nested exception is io.camunda.operate.exceptions.OperateRuntimeException: 
Error(s) in index schema: Newer version(s) for user (1.0.0) already exists: [1.2.0];
Newer version(s) for list-view (1.1.0) already exists: [1.2.0]

what can I do?
(I would like to keep the generated test data in the Zeebe Cluster itself, since this is from a performance test and it took quite a while to generate it).

Hi @nitram509 ,

the reason of your initial failure is that each Operate version is able to process data from the same minor version of Zeebe or the previous minor version of Zeebe. In your case data from Zeebe 1.0.5 could be used in Operate 1.0.x or 1.1.x. When you ran 1.2.5 you got the initial error about import not possible. Though it looks like that Operate created the empty indices for version 1.2.5 already, that’s why you get the second error.

In your current state I would propose you to remove all operate indices (by default they have operate prefix) and run Operate of version 1.1.7 (latest 1.1.x, it must be able to import data from version 1.0.x). After it imports Zeebe data, you can then upgrade further to any later version.

Does this help you?

Best regards,
Svetlana

2 Likes

Sounds reasonable … thank you Svetlana, will give it a try.

Hi Svetlana,

the attempt of upgrading/migrating to 1.1.7 first, seems to result in errors.
There are many more of them in the log … will wait some time, to see where it finally ends up.

2021-12-14 11:02:04.650  WARN 8 --- [           main] i.c.o.e.ElasticsearchConnector           : Elasticsearch cluster is not accessible
2021-12-14 11:02:04.746  INFO 8 --- [           main] i.c.o.w.StartupBean                      : INIT: Create users in elasticsearch if not exists ...
2021-12-14 11:02:04.803  INFO 8 --- [           main] .o.w.s.e.ElasticSearchUserDetailsService : Create user in ElasticSearch for username lynqtech
2021-12-14 11:02:04.989  INFO 8 --- [           main] .o.w.s.e.ElasticSearchUserDetailsService : Create user in ElasticSearch for username act
2021-12-14 11:02:05.098  INFO 8 --- [           main] i.c.o.w.StartupBean                      : INIT: Start operation executor...
2021-12-14 11:02:05.098  INFO 8 --- [           main] i.c.o.w.StartupBean                      : INIT: DONE
2021-12-14 11:02:05.403  INFO 8 --- [       Thread-1] i.c.o.z.ZeebeImporter                    : INIT: Start importing data...
2021-12-14 11:02:06.245  INFO 8 --- [           main] i.c.o.a.Archiver                         : INIT: Start archiving data...
2021-12-14 11:02:06.858  INFO 8 --- [       Thread-1] i.c.o.z.RecordsReaderHolder              : Starting import for partitions: [1, 2, 3]
2021-12-14 11:02:06.859  INFO 8 --- [           main] i.c.o.a.Archiver                         : Starting archiver for partitions: [1, 2, 3]
2021-12-14 11:02:08.923  INFO 8 --- [           main] o.s.b.a.e.w.EndpointLinksResolver        : Exposing 2 endpoint(s) beneath base path '/actuator'
2021-12-14 11:02:09.618 ERROR 8 --- [       import_2] i.c.o.z.ImportJob                        : Import is not possible for Zeebe version: 1.2
io.camunda.operate.exceptions.OperateRuntimeException: Import is not possible for Zeebe version: 1.2
	at io.camunda.operate.zeebeimport.ImportBatchProcessorFactory.getImportBatchProcessor(ImportBatchProcessorFactory.java:40) ~[operate-importer-1.1.7.jar!/:?]
	at io.camunda.operate.zeebeimport.ImportJob.processOneIndexBatch(ImportJob.java:115) ~[operate-importer-1.1.7.jar!/:?]
	at io.camunda.operate.zeebeimport.ImportJob.call(ImportJob.java:80) ~[operate-importer-1.1.7.jar!/:?]
	at io.camunda.operate.zeebeimport.RecordsReader.lambda$scheduleImport$1(RecordsReader.java:217) ~[operate-importer-1.1.7.jar!/:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
	at java.lang.Thread.run(Unknown Source) [?:?]
2021-12-14 11:02:11.206  INFO 8 --- [           main] o.s.s.w.DefaultSecurityFilterChain       : Will secure any request with [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@101ca8da, org.springframework.security.web.context.SecurityContextPersistenceFilter@32eae6f2, org.springframework.security.web.header.HeaderWriterFilter@16736040, org.springframework.security.web.csrf.CsrfFilter@68dd39d2, io.camunda.operate.webapp.security.CSRFProtectable$1@20a967fe, org.springframework.security.web.authentication.logout.LogoutFilter@6a5e167a, org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter@70721c12, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@73a0f2b, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@483b0690, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@405b6d75, org.springframework.security.web.session.SessionManagementFilter@550e9be6, org.springframework.security.web.access.ExceptionTranslationFilter@10272bbb, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@2582b0ef]
2021-12-14 11:02:12.039  INFO 8 --- [           main] o.s.b.a.w.s.WelcomePageHandlerMapping    : Adding welcome page: class path resource [META-INF/resources/index.html]
2021-12-14 11:02:12.707 ERROR 8 --- [       import_2] i.c.o.z.ImportJob                        : Import is not possible for Zeebe version: 1.2
io.camunda.operate.exceptions.OperateRuntimeException: Import is not possible for Zeebe version: 1.2
	at io.camunda.operate.zeebeimport.ImportBatchProcessorFactory.getImportBatchProcessor(ImportBatchProcessorFactory.java:40) ~[operate-importer-1.1.7.jar!/:?]
	at io.camunda.operate.zeebeimport.ImportJob.processOneIndexBatch(ImportJob.java:115) ~[operate-importer-1.1.7.jar!/:?]
	at io.camunda.operate.zeebeimport.ImportJob.call(ImportJob.java:80) ~[operate-importer-1.1.7.jar!/:?]
	at io.camunda.operate.zeebeimport.RecordsReader.lambda$scheduleImport$1(RecordsReader.java:217) ~[operate-importer-1.1.7.jar!/:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
	at java.lang.Thread.run(Unknown Source) [?:?]
2021-12-14 11:02:13.222  INFO 8 --- [           main] o.s.l.c.s.AbstractContextSource          : Property 'userDn' not set - anonymous context will be used for read-write operations
2021-12-14 11:02:14.528  INFO 8 --- [           main] o.a.c.h.Http11NioProtocol                : Starting ProtocolHandler ["http-nio-8080"]
2021-12-14 11:02:14.550  INFO 8 --- [           main] o.s.b.w.e.t.TomcatWebServer              : Tomcat started on port(s): 8080 (http) with context path ''
2021-12-14 11:02:18.217  INFO 8 --- [           main] i.c.o.Application                        : Started Application in 34.465 seconds (JVM running for 38.988)
2021-12-14 11:02:18.226  INFO 8 --- [           main] i.c.o.s.m.SchemaMigration                : SchemaMigration finished.

Hm, you’re now running Zeebe of v. 1.2.x already?

It looks like Operate 1.1.7 imported all Zeebe data of versions 1.0.x and 1.1.x and now fails on 1.2.x, which makes sense. Try now to update Operate to latest 1.2.4, it must catch up and continue processing data of version 1.2.x.

1 Like

Thank you for your reply.
Indeed, I left the Zeebe cluster as is, means, I came from 1.0.x and went straight to 1.2.x.
And since test data already generated, I did not drop all indexes.

Unfortunately, the log is still full of errors …

[           main] i.c.o.s.SchemaStartup                    : SchemaStartup started.
[           main] i.c.o.s.SchemaStartup                    : SchemaStartup: validate schema.
[           main] i.c.o.s.SchemaStartup                    : SchemaStartup: schema is empty or not complete. Indices will be created.
[           main] i.c.o.s.ElasticsearchSchemaManager       : Create default settings from 'operate_template' with 1 shards and 0 replicas per index.
[           main] i.c.o.s.SchemaStartup                    : SchemaStartup: migrate schema.
[           main] i.c.o.s.m.ElasticsearchStepsRepository   : Read step 1.2.0-1_incident_script.json 
[           main] i.c.o.s.m.ElasticsearchStepsRepository   : Read step 1.2.0-3_user_script.json 
[           main] i.c.o.s.m.ElasticsearchStepsRepository   : Read step 1.2.0-4_user_script.json 
[           main] i.c.o.s.m.ElasticsearchStepsRepository   : Read step 1.2.0-2_user_script.json 
[           main] i.c.o.s.m.ElasticsearchStepsRepository   : Read step 1.2.0-5_user_delete-fields_script.json 
[           main] i.c.o.s.m.ElasticsearchStepsRepository   : Read step 1.2.0-0_list-view_script.json 
[           main] i.c.o.s.m.ElasticsearchStepsRepository   : Read step 1.1.0-0_variable_script.json 
[           main] i.c.o.s.m.ElasticsearchStepsRepository   : Add new step ProcessorStep [content={"script": {"lang": "painless","source": "if (ctx.joinRelation.name.equals('processInstance')) { ctx.treePath = 'PI_' + ctx.processInstanceKey; }"}}, appliedDate=null, indexName=list-view, isApplied=false, version=1.2.0, order=0, createdDate=2021-12-14T13:33:31.340588Z] to repository.
[           main] i.c.o.s.m.ElasticsearchStepsRepository   : Step ProcessorStep [content={"script": {"lang": "painless","source": "if (ctx.joinRelation.name.equals('processInstance')) { ctx.treePath = 'PI_' + ctx.processInstanceKey; }"}}, appliedDate=null, indexName=list-view, isApplied=false, version=1.2.0, order=0, createdDate=2021-12-14T13:33:31.340588Z]  saved.
[           main] i.c.o.s.m.ElasticsearchStepsRepository   : Add new step ProcessorStep [content={"script": {"lang": "painless","source": "ctx.treePath = 'PI_' + ctx.processInstanceKey + '/FN_' + ctx.flowNodeId + '/FNI_' + ctx.flowNodeInstanceKey;"}}, appliedDate=null, indexName=incident, isApplied=false, version=1.2.0, order=1, createdDate=2021-12-14T13:33:31.442858Z] to repository.
[           main] i.c.o.s.m.ElasticsearchStepsRepository   : Step ProcessorStep [content={"script": {"lang": "painless","source": "ctx.treePath = 'PI_' + ctx.processInstanceKey + '/FN_' + ctx.flowNodeId + '/FNI_' + ctx.flowNodeInstanceKey;"}}, appliedDate=null, indexName=incident, isApplied=false, version=1.2.0, order=1, createdDate=2021-12-14T13:33:31.442858Z]  saved.
[           main] i.c.o.s.m.ElasticsearchStepsRepository   : Add new step ProcessorStep [content={"remove": {"field": "role" }}, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=2, createdDate=2021-12-14T13:33:31.452793Z] to repository.
[           main] i.c.o.s.m.ElasticsearchStepsRepository   : Step ProcessorStep [content={"remove": {"field": "role" }}, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=2, createdDate=2021-12-14T13:33:31.452793Z]  saved.
[           main] i.c.o.s.m.ElasticsearchStepsRepository   : Add new step ProcessorStep [content={"append": {"field": "roles", "value": ["OWNER"]}}, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=3, createdDate=2021-12-14T13:33:31.462780Z] to repository.
[           main] i.c.o.s.m.ElasticsearchStepsRepository   : Step ProcessorStep [content={"append": {"field": "roles", "value": ["OWNER"]}}, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=3, createdDate=2021-12-14T13:33:31.462780Z]  saved.
[           main] i.c.o.s.m.ElasticsearchStepsRepository   : Add new step ProcessorStep [content={"script": {"lang": "painless","source": "ctx.userId = ctx.username; ctx.displayName = (ctx.firstname == null?'':ctx.firstname) + (ctx.lastname == null?'': (' ' + ctx.lastname))"}}, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=4, createdDate=2021-12-14T13:33:31.473764Z] to repository.
[           main] i.c.o.s.m.ElasticsearchStepsRepository   : Step ProcessorStep [content={"script": {"lang": "painless","source": "ctx.userId = ctx.username; ctx.displayName = (ctx.firstname == null?'':ctx.firstname) + (ctx.lastname == null?'': (' ' + ctx.lastname))"}}, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=4, createdDate=2021-12-14T13:33:31.473764Z]  saved.
[           main] i.c.o.s.m.ElasticsearchStepsRepository   : Add new step ProcessorStep [content={ "remove": { "field": ["username", "firstname", "lastname"] } }, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=5, createdDate=2021-12-14T13:33:31.482649Z] to repository.
[           main] i.c.o.s.m.ElasticsearchStepsRepository   : Step ProcessorStep [content={ "remove": { "field": ["username", "firstname", "lastname"] } }, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=5, createdDate=2021-12-14T13:33:31.482649Z]  saved.
[    migration_1] i.c.o.s.m.Migrator                       : Check if index migration-steps-repository needs to migrate.
[    migration_5] i.c.o.s.m.Migrator                       : Check if index process needs to migrate.
[    migration_3] i.c.o.s.m.Migrator                       : Check if index import-position needs to migrate.
[    migration_2] i.c.o.s.m.Migrator                       : Check if index user needs to migrate.
[    migration_4] i.c.o.s.m.Migrator                       : Check if index web-session needs to migrate.
[    migration_1] i.c.o.s.m.Migrator                       : No migration needed for migration-steps-repository, no previous indices found.
[    migration_1] i.c.o.s.m.Migrator                       : Check if index sequence-flow needs to migrate.
[    migration_1] i.c.o.s.m.Migrator                       : No migration needed for sequence-flow, no previous indices found.
[    migration_1] i.c.o.s.m.Migrator                       : Check if index incident needs to migrate.
[    migration_4] i.c.o.s.m.Migrator                       : No migration needed for web-session, no previous indices found.
[    migration_3] i.c.o.s.m.Migrator                       : No migration needed for import-position, no previous indices found.
[    migration_3] i.c.o.s.m.Migrator                       : Check if index batch-operation needs to migrate.
[    migration_4] i.c.o.s.m.Migrator                       : Check if index variable needs to migrate.
[    migration_5] i.c.o.s.m.Migrator                       : No migration needed for process, no previous indices found.
[    migration_5] i.c.o.s.m.Migrator                       : Check if index event needs to migrate.
[    migration_3] i.c.o.s.m.Migrator                       : No migration needed for batch-operation, no previous indices found.
[    migration_3] i.c.o.s.m.Migrator                       : Check if index flownode-instance needs to migrate.
[    migration_3] i.c.o.s.m.Migrator                       : No migration needed for flownode-instance, no previous indices found.
[    migration_3] i.c.o.s.m.Migrator                       : Check if index list-view needs to migrate.
[    migration_5] i.c.o.s.m.Migrator                       : No migration needed for event, no previous indices found.
[    migration_5] i.c.o.s.m.Migrator                       : Check if index operation needs to migrate.
[    migration_5] i.c.o.s.m.Migrator                       : No migration needed for operation, no previous indices found.
[    migration_4] i.c.o.s.m.Migrator                       : No migration needed for variable, no previous indices found.
[    migration_1] i.c.o.s.m.Migrator                       : Execute plan: ReindexPlan [steps=[ProcessorStep [content={"script": {"lang": "painless","source": "ctx.treePath = 'PI_' + ctx.processInstanceKey + '/FN_' + ctx.flowNodeId + '/FNI_' + ctx.flowNodeInstanceKey;"}}, appliedDate=null, indexName=incident, isApplied=false, version=1.2.0, order=1, createdDate=2021-12-14T13:33:31.442Z]],  srcIndex=operate-incident-1.0.0, dstIndex=operate-incident-1.2.0] 
[    migration_3] i.c.o.s.m.Migrator                       : Execute plan: ReindexPlan [steps=[ProcessorStep [content={"script": {"lang": "painless","source": "if (ctx.joinRelation.name.equals('processInstance')) { ctx.treePath = 'PI_' + ctx.processInstanceKey; }"}}, appliedDate=null, indexName=list-view, isApplied=false, version=1.2.0, order=0, createdDate=2021-12-14T13:33:31.340Z]],  srcIndex=operate-list-view-1.1.0, dstIndex=operate-list-view-1.2.0] 
[    migration_2] i.c.o.s.m.Migrator                       : Execute plan: ReindexPlan [steps=[ProcessorStep [content={"remove": {"field": "role" }}, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=2, createdDate=2021-12-14T13:33:31.452Z], ProcessorStep [content={"append": {"field": "roles", "value": ["OWNER"]}}, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=3, createdDate=2021-12-14T13:33:31.462Z], ProcessorStep [content={"script": {"lang": "painless","source": "ctx.userId = ctx.username; ctx.displayName = (ctx.firstname == null?'':ctx.firstname) + (ctx.lastname == null?'': (' ' + ctx.lastname))"}}, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=4, createdDate=2021-12-14T13:33:31.473Z], ProcessorStep [content={ "remove": { "field": ["username", "firstname", "lastname"] } }, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=5, createdDate=2021-12-14T13:33:31.482Z]],  srcIndex=operate-user-1.0.0, dstIndex=operate-user-1.2.0] 
[    migration_2] i.c.o.e.RetryElasticsearchClient         : Source docs: 2, Migrated docs: 2
[    migration_2] i.c.o.s.m.ElasticsearchStepsRepository   : Step ProcessorStep [content={"remove": {"field": "role" }}, appliedDate=2021-12-14T13:33:35.819553Z, indexName=user, isApplied=true, version=1.2.0, order=2, createdDate=2021-12-14T13:33:31.452Z]  saved.
[    migration_2] i.c.o.s.m.ElasticsearchStepsRepository   : Step ProcessorStep [content={"append": {"field": "roles", "value": ["OWNER"]}}, appliedDate=2021-12-14T13:33:35.852049Z, indexName=user, isApplied=true, version=1.2.0, order=3, createdDate=2021-12-14T13:33:31.462Z]  saved.
[    migration_2] i.c.o.s.m.ElasticsearchStepsRepository   : Step ProcessorStep [content={"script": {"lang": "painless","source": "ctx.userId = ctx.username; ctx.displayName = (ctx.firstname == null?'':ctx.firstname) + (ctx.lastname == null?'': (' ' + ctx.lastname))"}}, appliedDate=2021-12-14T13:33:35.865073Z, indexName=user, isApplied=true, version=1.2.0, order=4, createdDate=2021-12-14T13:33:31.473Z]  saved.
[    migration_2] i.c.o.s.m.ElasticsearchStepsRepository   : Step ProcessorStep [content={ "remove": { "field": ["username", "firstname", "lastname"] } }, appliedDate=2021-12-14T13:33:35.889153Z, indexName=user, isApplied=true, version=1.2.0, order=5, createdDate=2021-12-14T13:33:31.482Z]  saved.
[    migration_1] i.c.o.e.RetryElasticsearchClient         : Source docs: 17, Migrated docs: 17
[    migration_2] i.c.o.s.m.Migrator                       : Refresh index operate-user-1.2.0_*
[    migration_2] i.c.o.s.m.Migrator                       : Deleted previous indices for pattern operate-user-1.0.0_*
[    migration_1] i.c.o.s.m.ElasticsearchStepsRepository   : Step ProcessorStep [content={"script": {"lang": "painless","source": "ctx.treePath = 'PI_' + ctx.processInstanceKey + '/FN_' + ctx.flowNodeId + '/FNI_' + ctx.flowNodeInstanceKey;"}}, appliedDate=2021-12-14T13:33:36.202377Z, indexName=incident, isApplied=true, version=1.2.0, order=1, createdDate=2021-12-14T13:33:31.442Z]  saved.
[    migration_1] i.c.o.s.m.Migrator                       : Refresh index operate-incident-1.2.0_*
[    migration_1] i.c.o.s.m.Migrator                       : Deleted previous indices for pattern operate-incident-1.0.0_*
[    migration_1] i.c.o.s.m.Migrator                       : Deleted previous templates for operate-incident-1.0.0_template
[    migration_3] i.c.o.e.RetryElasticsearchClient         : Source docs: 61059, Migrated docs: 61059
[    migration_3] i.c.o.s.m.ElasticsearchStepsRepository   : Step ProcessorStep [content={"script": {"lang": "painless","source": "if (ctx.joinRelation.name.equals('processInstance')) { ctx.treePath = 'PI_' + ctx.processInstanceKey; }"}}, appliedDate=2021-12-14T13:34:12.156130Z, indexName=list-view, isApplied=true, version=1.2.0, order=0, createdDate=2021-12-14T13:33:31.340Z]  saved.
[    migration_3] i.c.o.s.m.Migrator                       : Refresh index operate-list-view-1.2.0_*
[    migration_3] i.c.o.s.m.Migrator                       : Deleted previous indices for pattern operate-list-view-1.1.0_*
[    migration_3] i.c.o.s.m.Migrator                       : Deleted previous templates for operate-list-view-1.1.0_template
[           main] i.c.o.s.SchemaStartup                    : SchemaStartup finished.
[           main] i.c.o.ImportModuleConfiguration          : Starting module: importer
[           main] i.c.o.ArchiverModuleConfiguration        : Starting module: archiver
2021-12-14 13:34:13.840  WARN 8 --- [           main] i.c.o.e.ElasticsearchConnector           : Username and/or password for are empty. Basic authentication for elasticsearch is not used.
2021-12-14 13:34:13.945  WARN 8 --- [           main] i.c.o.e.ElasticsearchConnector           : Elasticsearch cluster is not accessible
[           main] i.c.o.w.StartupBean                      : INIT: Create users in elasticsearch if not exists ...
[           main] .o.w.s.e.ElasticSearchUserDetailsService : Create user in ElasticSearch for userId demo
[           main] .o.w.s.e.ElasticSearchUserDetailsService : Create user in ElasticSearch for userId view
[           main] i.c.o.w.StartupBean                      : INIT: Start operation executor...
[           main] i.c.o.w.StartupBean                      : INIT: DONE
[       Thread-1] i.c.o.z.ZeebeImporter                    : INIT: Start importing data...
[           main] i.c.o.a.Archiver                         : INIT: Start archiving data...
[           main] i.c.o.a.Archiver                         : Starting archiver for partitions: [1, 2, 3]
[       Thread-1] i.c.o.z.RecordsReaderHolder              : Starting import for partitions: [1, 2, 3]
2021-12-14 13:34:17.025 ERROR 8 --- [       import_1] i.c.o.z.ImportJob                        : Import is not possible for Zeebe version: 1.0
io.camunda.operate.exceptions.OperateRuntimeException: Import is not possible for Zeebe version: 1.0
	at io.camunda.operate.zeebeimport.ImportBatchProcessorFactory.getImportBatchProcessor(ImportBatchProcessorFactory.java:40) ~[operate-importer-1.2.6.jar!/:?]
	at io.camunda.operate.zeebeimport.ImportJob.processOneIndexBatch(ImportJob.java:115) ~[operate-importer-1.2.6.jar!/:?]
	at io.camunda.operate.zeebeimport.ImportJob.call(ImportJob.java:80) ~[operate-importer-1.2.6.jar!/:?]
	at io.camunda.operate.zeebeimport.RecordsReader.lambda$scheduleImport$1(RecordsReader.java:217) ~[operate-importer-1.2.6.jar!/:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
	at java.lang.Thread.run(Unknown Source) [?:?]
2021-12-14 13:34:17.370 ERROR 8 --- [       import_2] i.c.o.z.ImportJob                        : Import is not possible for Zeebe version: 1.0
io.camunda.operate.exceptions.OperateRuntimeException: Import is not possible for Zeebe version: 1.0
	at io.camunda.operate.zeebeimport.ImportBatchProcessorFactory.getImportBatchProcessor(ImportBatchProcessorFactory.java:40) ~[operate-importer-1.2.6.jar!/:?]
	at io.camunda.operate.zeebeimport.ImportJob.processOneIndexBatch(ImportJob.java:115) ~[operate-importer-1.2.6.jar!/:?]
	at io.camunda.operate.zeebeimport.ImportJob.call(ImportJob.java:80) ~[operate-importer-1.2.6.jar!/:?]
	at io.camunda.operate.zeebeimport.RecordsReader.lambda$scheduleImport$1(RecordsReader.java:217) ~[operate-importer-1.2.6.jar!/:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
	at java.lang.Thread.run(Unknown Source) [?:?]
2021-12-14 13:34:17.942 ERROR 8 --- [       import_3] i.c.o.z.ImportJob                        : Import is not possible for Zeebe version: 1.0
io.camunda.operate.exceptions.OperateRuntimeException: Import is not possible for Zeebe version: 1.0
	at io.camunda.operate.zeebeimport.ImportBatchProcessorFactory.getImportBatchProcessor(ImportBatchProcessorFactory.java:40) ~[operate-importer-1.2.6.jar!/:?]
	at io.camunda.operate.zeebeimport.ImportJob.processOneIndexBatch(ImportJob.java:115) ~[operate-importer-1.2.6.jar!/:?]
	at io.camunda.operate.zeebeimport.ImportJob.call(ImportJob.java:80) ~[operate-importer-1.2.6.jar!/:?]
	at io.camunda.operate.zeebeimport.RecordsReader.lambda$scheduleImport$1(RecordsReader.java:217) ~[operate-importer-1.2.6.jar!/:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
	at java.lang.Thread.run(Unknown Source) [?:?]
[           main] o.s.b.a.e.w.EndpointLinksResolver        : Exposing 2 endpoint(s) beneath base path '/actuator'
2021-12-14 13:34:19.036 ERROR 8 --- [       import_1] i.c.o.z.ImportJob                        : Import is not possible for Zeebe version: 1.0
io.camunda.operate.exceptions.OperateRuntimeException: Import is not possible for Zeebe version: 1.0
	at io.camunda.operate.zeebeimport.ImportBatchProcessorFactory.getImportBatchProcessor(ImportBatchProcessorFactory.java:40) ~[operate-importer-1.2.6.jar!/:?]
	at io.camunda.operate.zeebeimport.ImportJob.processOneIndexBatch(ImportJob.java:115) ~[operate-importer-1.2.6.jar!/:?]
	at io.camunda.operate.zeebeimport.ImportJob.call(ImportJob.java:80) ~[operate-importer-1.2.6.jar!/:?]
	at io.camunda.operate.zeebeimport.RecordsReader.lambda$scheduleImport$1(RecordsReader.java:217) ~[operate-importer-1.2.6.jar!/:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
	at java.lang.Thread.run(Unknown Source) [?:?]
[           main] o.s.s.w.DefaultSecurityFilterChain       : Will secure any request with [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@6097fca9, org.springframework.security.web.context.SecurityContextPersistenceFilter@5a566922, org.springframework.security.web.header.HeaderWriterFilter@bc09d57, org.springframework.security.web.csrf.CsrfFilter@59d0cdb6, io.camunda.operate.webapp.security.BaseWebConfigurer$1@35eee641, org.springframework.security.web.authentication.logout.LogoutFilter@66749eea, org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter@eed890d, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@42ed89da, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@5e180aaf, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@5729b410, org.springframework.security.web.session.SessionManagementFilter@467045c4, org.springframework.security.web.access.ExceptionTranslationFilter@4d0cc83e, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@7aaad0]
2021-12-14 13:34:19.375 ERROR 8 --- [       import_2] i.c.o.z.ImportJob                        : Import is not possible for Zeebe version: 1.0
io.camunda.operate.exceptions.OperateRuntimeException: Import is not possible for Zeebe version: 1.0
	at io.camunda.operate.zeebeimport.ImportBatchProcessorFactory.getImportBatchProcessor(ImportBatchProcessorFactory.java:40) ~[operate-importer-1.2.6.jar!/:?]
	at io.camunda.operate.zeebeimport.ImportJob.processOneIndexBatch(ImportJob.java:115) ~[operate-importer-1.2.6.jar!/:?]
	at io.camunda.operate.zeebeimport.ImportJob.call(ImportJob.java:80) ~[operate-importer-1.2.6.jar!/:?]
	at io.camunda.operate.zeebeimport.RecordsReader.lambda$scheduleImport$1(RecordsReader.java:217) ~[operate-importer-1.2.6.jar!/:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
	at java.lang.Thread.run(Unknown Source) [?:?]
[           main] o.h.v.i.u.Version                        : HV000001: Hibernate Validator 6.2.0.Final
2021-12-14 13:34:19.978 ERROR 8 --- [       import_3] i.c.o.z.ImportJob                        : Import is not possible for Zeebe version: 1.0
io.camunda.operate.exceptions.OperateRuntimeException: Import is not possible for Zeebe version: 1.0
	at io.camunda.operate.zeebeimport.ImportBatchProcessorFactory.getImportBatchProcessor(ImportBatchProcessorFactory.java:40) ~[operate-importer-1.2.6.jar!/:?]
	at io.camunda.operate.zeebeimport.ImportJob.processOneIndexBatch(ImportJob.java:115) ~[operate-importer-1.2.6.jar!/:?]
	at io.camunda.operate.zeebeimport.ImportJob.call(ImportJob.java:80) ~[operate-importer-1.2.6.jar!/:?]
	at io.camunda.operate.zeebeimport.RecordsReader.lambda$scheduleImport$1(RecordsReader.java:217) ~[operate-importer-1.2.6.jar!/:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
	at java.lang.Thread.run(Unknown Source) [?:?]
[           main] o.s.b.a.w.s.WelcomePageHandlerMapping    : Adding welcome page: class path resource [META-INF/resources/index.html]
[           main] o.s.l.c.s.AbstractContextSource          : Property 'userDn' not set - anonymous context will be used for read-write operations
2021-12-14 13:34:21.243 ERROR 8 --- [       import_1] i.c.o.z.ImportJob                        : Import is not possible for Zeebe version: 1.0
io.camunda.operate.exceptions.OperateRuntimeException: Import is not possible for Zeebe version: 1.0
	at io.camunda.operate.zeebeimport.ImportBatchProcessorFactory.getImportBatchProcessor(ImportBatchProcessorFactory.java:40) ~[operate-importer-1.2.6.jar!/:?]
	at io.camunda.operate.zeebeimport.ImportJob.processOneIndexBatch(ImportJob.java:115) ~[operate-importer-1.2.6.jar!/:?]
	at io.camunda.operate.zeebeimport.ImportJob.call(ImportJob.java:80) ~[operate-importer-1.2.6.jar!/:?]
	at io.camunda.operate.zeebeimport.RecordsReader.lambda$scheduleImport$1(RecordsReader.java:217) ~[operate-importer-1.2.6.jar!/:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
	at java.lang.Thread.run(Unknown Source) [?:?]

Any hints?

Hi Svetlana,

there’s one observation I would like to add…

With my “two-step-approach” (see above), I was able to get some process information visible in Operate.
Unfortunately, I was not able to get any updated data visible (waited ca. 1h but no new instance showed up).
Also, I realized a drastic load increase on elasticsearch search rate:
normal load: ca. 40 search req/min
with Operate running (and massively complaining about migration issues): ca. 1800 search req/min
This did not go down within ca. 2hours.
(number of documents is ca. 300k)

Hi @nitram509 ,
let’s try to understand what’s the current state of your data. Could you share the current list of indices that you have in Elastic and the content of operate_import-position_* index?

Hi Svetlana,

here’s the list of all indexes (alphabetical order).

And also all documents from the operate-import-position-1.0.0_ index (below)

operate-batch-operation-1.0.0_
operate-event-1.0.0_
operate-event-1.0.0_2021-11-08
operate-event-1.0.0_2021-11-30
operate-event-1.0.0_2021-12-01
operate-flownode-instance-1.0.0_
operate-flownode-instance-1.0.0_2021-11-08
operate-flownode-instance-1.0.0_2021-11-30
operate-flownode-instance-1.0.0_2021-12-01
operate-import-position-1.0.0_
operate-incident-1.2.0_
operate-list-view-1.2.0_
operate-list-view-1.2.0_2021-11-08
operate-list-view-1.2.0_2021-11-30
operate-list-view-1.2.0_2021-12-01
operate-migration-steps-repository-1.1.0_
operate-operation-1.0.0_
operate-process-1.0.0_
operate-sequence-flow-1.0.0_
operate-sequence-flow-1.0.0_2021-11-08
operate-sequence-flow-1.0.0_2021-11-30
operate-sequence-flow-1.0.0_2021-12-01
operate-user-1.2.0_
operate-variable-1.1.0_
operate-variable-1.1.0_2021-11-08
operate-variable-1.1.0_2021-11-30
operate-variable-1.1.0_2021-12-01
operate-web-session-1.1.0_

zeebe-record_incident_1.0.5_2021-11-06
zeebe-record_incident_1.0.5_2021-11-09
zeebe-record_incident_1.0.5_2021-11-11
zeebe-record_incident_1.0.5_2021-11-13
zeebe-record_incident_1.0.5_2021-11-16
zeebe-record_incident_1.0.5_2021-11-17
zeebe-record_incident_1.0.5_2021-11-18
zeebe-record_incident_1.0.5_2021-11-19
zeebe-record_incident_1.0.5_2021-11-25
zeebe-record_incident_1.0.5_2021-11-26
zeebe-record_incident_1.0.5_2021-11-27
zeebe-record_incident_1.0.5_2021-11-28
zeebe-record_incident_1.0.5_2021-11-29
zeebe-record_incident_1.0.5_2021-11-30
zeebe-record_incident_1.0.5_2021-12-01
zeebe-record_incident_1.0.5_2021-12-02
zeebe-record_incident_1.0.5_2021-12-03
zeebe-record_incident_1.0.5_2021-12-04
zeebe-record_incident_1.0.5_2021-12-05
zeebe-record_incident_1.0.5_2021-12-06
zeebe-record_incident_1.2.4_2021-12-07
zeebe-record_incident_1.2.4_2021-12-08
zeebe-record_incident_1.2.4_2021-12-09
zeebe-record_incident_1.2.4_2021-12-10
zeebe-record_incident_1.2.4_2021-12-11
zeebe-record_incident_1.2.4_2021-12-12
zeebe-record_incident_1.2.4_2021-12-13
zeebe-record_incident_1.2.6_2021-12-14
zeebe-record_incident_1.2.6_2021-12-15
zeebe-record_job_1.0.5_2021-11-05
zeebe-record_job_1.0.5_2021-11-06
zeebe-record_job_1.0.5_2021-11-08
zeebe-record_job_1.0.5_2021-11-09
zeebe-record_job_1.0.5_2021-11-10
zeebe-record_job_1.0.5_2021-11-11
zeebe-record_job_1.0.5_2021-11-12
zeebe-record_job_1.0.5_2021-11-13
zeebe-record_job_1.0.5_2021-11-15
zeebe-record_job_1.0.5_2021-11-16
zeebe-record_job_1.0.5_2021-11-17
zeebe-record_job_1.0.5_2021-11-18
zeebe-record_job_1.0.5_2021-11-19
zeebe-record_job_1.0.5_2021-11-20
zeebe-record_job_1.0.5_2021-11-21
zeebe-record_job_1.0.5_2021-11-22
zeebe-record_job_1.0.5_2021-11-23
zeebe-record_job_1.0.5_2021-11-24
zeebe-record_job_1.0.5_2021-11-25
zeebe-record_job_1.0.5_2021-11-26
zeebe-record_job_1.0.5_2021-11-27
zeebe-record_job_1.0.5_2021-11-28
zeebe-record_job_1.0.5_2021-11-29
zeebe-record_job_1.0.5_2021-11-30
zeebe-record_job_1.0.5_2021-12-01
zeebe-record_job_1.0.5_2021-12-02
zeebe-record_job_1.0.5_2021-12-03
zeebe-record_job_1.0.5_2021-12-04
zeebe-record_job_1.0.5_2021-12-05
zeebe-record_job_1.0.5_2021-12-06
zeebe-record_job_1.0.5_2021-12-07
zeebe-record_job_1.2.4_2021-12-07
zeebe-record_job_1.2.4_2021-12-08
zeebe-record_job_1.2.4_2021-12-09
zeebe-record_job_1.2.4_2021-12-10
zeebe-record_job_1.2.4_2021-12-11
zeebe-record_job_1.2.4_2021-12-12
zeebe-record_job_1.2.4_2021-12-13
zeebe-record_job_1.2.6_2021-12-14
zeebe-record_job_1.2.6_2021-12-15
zeebe-record_process_1.0.5_2021-11-05
zeebe-record_process_1.0.5_2021-11-10
zeebe-record_process_1.0.5_2021-11-12
zeebe-record_process_1.0.5_2021-11-25
zeebe-record_process_1.0.5_2021-11-30
zeebe-record_process_1.0.5_2021-12-01
zeebe-record_process_1.2.4_2021-12-09
zeebe-record_process_1.2.6_2021-12-15
zeebe-record_process-instance_1.0.5_2021-11-05
zeebe-record_process-instance_1.0.5_2021-11-06
zeebe-record_process-instance_1.0.5_2021-11-08
zeebe-record_process-instance_1.0.5_2021-11-09
zeebe-record_process-instance_1.0.5_2021-11-10
zeebe-record_process-instance_1.0.5_2021-11-11
zeebe-record_process-instance_1.0.5_2021-11-12
zeebe-record_process-instance_1.0.5_2021-11-13
zeebe-record_process-instance_1.0.5_2021-11-15
zeebe-record_process-instance_1.0.5_2021-11-16
zeebe-record_process-instance_1.0.5_2021-11-17
zeebe-record_process-instance_1.0.5_2021-11-18
zeebe-record_process-instance_1.0.5_2021-11-19
zeebe-record_process-instance_1.0.5_2021-11-20
zeebe-record_process-instance_1.0.5_2021-11-21
zeebe-record_process-instance_1.0.5_2021-11-22
zeebe-record_process-instance_1.0.5_2021-11-23
zeebe-record_process-instance_1.0.5_2021-11-24
zeebe-record_process-instance_1.0.5_2021-11-25
zeebe-record_process-instance_1.0.5_2021-11-26
zeebe-record_process-instance_1.0.5_2021-11-27
zeebe-record_process-instance_1.0.5_2021-11-28
zeebe-record_process-instance_1.0.5_2021-11-29
zeebe-record_process-instance_1.0.5_2021-11-30
zeebe-record_process-instance_1.0.5_2021-12-01
zeebe-record_process-instance_1.0.5_2021-12-02
zeebe-record_process-instance_1.0.5_2021-12-03
zeebe-record_process-instance_1.0.5_2021-12-04
zeebe-record_process-instance_1.0.5_2021-12-05
zeebe-record_process-instance_1.0.5_2021-12-06
zeebe-record_process-instance_1.0.5_2021-12-07
zeebe-record_process-instance_1.2.4_2021-12-07
zeebe-record_process-instance_1.2.4_2021-12-08
zeebe-record_process-instance_1.2.4_2021-12-09
zeebe-record_process-instance_1.2.4_2021-12-10
zeebe-record_process-instance_1.2.4_2021-12-11
zeebe-record_process-instance_1.2.4_2021-12-12
zeebe-record_process-instance_1.2.4_2021-12-13
zeebe-record_process-instance_1.2.6_2021-12-14
zeebe-record_process-instance_1.2.6_2021-12-15
zeebe-record_variable_1.0.5_2021-11-05
zeebe-record_variable_1.0.5_2021-11-08
zeebe-record_variable_1.0.5_2021-11-10
zeebe-record_variable_1.0.5_2021-11-11
zeebe-record_variable_1.0.5_2021-11-12
zeebe-record_variable_1.0.5_2021-11-15
zeebe-record_variable_1.0.5_2021-11-16
zeebe-record_variable_1.0.5_2021-11-17
zeebe-record_variable_1.0.5_2021-11-18
zeebe-record_variable_1.0.5_2021-11-19
zeebe-record_variable_1.0.5_2021-11-20
zeebe-record_variable_1.0.5_2021-11-21
zeebe-record_variable_1.0.5_2021-11-22
zeebe-record_variable_1.0.5_2021-11-23
zeebe-record_variable_1.0.5_2021-11-24
zeebe-record_variable_1.0.5_2021-11-25
zeebe-record_variable_1.0.5_2021-11-26
zeebe-record_variable_1.0.5_2021-11-27
zeebe-record_variable_1.0.5_2021-11-28
zeebe-record_variable_1.0.5_2021-11-29
zeebe-record_variable_1.0.5_2021-11-30
zeebe-record_variable_1.0.5_2021-12-01
zeebe-record_variable_1.0.5_2021-12-02
zeebe-record_variable_1.0.5_2021-12-03
zeebe-record_variable_1.0.5_2021-12-04
zeebe-record_variable_1.0.5_2021-12-05
zeebe-record_variable_1.0.5_2021-12-06
zeebe-record_variable_1.0.5_2021-12-07
zeebe-record_variable_1.2.4_2021-12-07
zeebe-record_variable_1.2.4_2021-12-08
zeebe-record_variable_1.2.4_2021-12-09
zeebe-record_variable_1.2.4_2021-12-10
zeebe-record_variable_1.2.4_2021-12-11
zeebe-record_variable_1.2.4_2021-12-12
zeebe-record_variable_1.2.4_2021-12-13
zeebe-record_variable_1.2.6_2021-12-14
zeebe-record_variable_1.2.6_2021-12-15

GET operate-import-position-1.0.0_/_search …

{
  "took" : 4,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 8,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "operate-import-position-1.0.0_",
        "_type" : "_doc",
        "_id" : "3-job",
        "_score" : 1.0,
        "_source" : {
          "id" : "3-job",
          "aliasName" : "job",
          "partitionId" : 3,
          "position" : 606808,
          "indexName" : "zeebe-record_job_1.0.5_2021-12-05"
        }
      },
      {
        "_index" : "operate-import-position-1.0.0_",
        "_type" : "_doc",
        "_id" : "2-job",
        "_score" : 1.0,
        "_source" : {
          "id" : "2-job",
          "aliasName" : "job",
          "partitionId" : 2,
          "position" : 605321,
          "indexName" : "zeebe-record_job_1.0.5_2021-12-05"
        }
      },
      {
        "_index" : "operate-import-position-1.0.0_",
        "_type" : "_doc",
        "_id" : "1-process-instance",
        "_score" : 1.0,
        "_source" : {
          "id" : "1-process-instance",
          "aliasName" : "process-instance",
          "partitionId" : 1,
          "position" : 737964,
          "indexName" : "zeebe-record_process-instance_1.0.5_2021-12-06"
        }
      },
      {
        "_index" : "operate-import-position-1.0.0_",
        "_type" : "_doc",
        "_id" : "1-variable",
        "_score" : 1.0,
        "_source" : {
          "id" : "1-variable",
          "aliasName" : "variable",
          "partitionId" : 1,
          "position" : 735755,
          "indexName" : "zeebe-record_variable_1.0.5_2021-12-06"
        }
      },
      {
        "_index" : "operate-import-position-1.0.0_",
        "_type" : "_doc",
        "_id" : "3-process-instance",
        "_score" : 1.0,
        "_source" : {
          "id" : "3-process-instance",
          "aliasName" : "process-instance",
          "partitionId" : 3,
          "position" : 726701,
          "indexName" : "zeebe-record_process-instance_1.0.5_2021-12-06"
        }
      },
      {
        "_index" : "operate-import-position-1.0.0_",
        "_type" : "_doc",
        "_id" : "2-variable",
        "_score" : 1.0,
        "_source" : {
          "id" : "2-variable",
          "aliasName" : "variable",
          "partitionId" : 2,
          "position" : 739881,
          "indexName" : "zeebe-record_variable_1.0.5_2021-12-06"
        }
      },
      {
        "_index" : "operate-import-position-1.0.0_",
        "_type" : "_doc",
        "_id" : "3-variable",
        "_score" : 1.0,
        "_source" : {
          "id" : "3-variable",
          "aliasName" : "variable",
          "partitionId" : 3,
          "position" : 763224,
          "indexName" : "zeebe-record_variable_1.0.5_2021-12-06"
        }
      },
      {
        "_index" : "operate-import-position-1.0.0_",
        "_type" : "_doc",
        "_id" : "2-process-instance",
        "_score" : 1.0,
        "_source" : {
          "id" : "2-process-instance",
          "aliasName" : "process-instance",
          "partitionId" : 2,
          "position" : 791887,
          "indexName" : "zeebe-record_process-instance_1.0.5_2021-12-06"
        }
      }
    ]
  }
}

Hi @nitram509 ,
thank you! That’s actually an interesting case. I think I understand now what’s happening. Zeebe indices are joined in Elastic alises, e.g. all zeebe-record_job_* indices belong to alias zeebe-record_job. Operate reads data in batches and from aliases, therefore it can happen that one batch contained data from 2 versions (e.g. from indices zeebe-record_job_1.0.5_2021-12-07 and zeebe-record_job_1.2.4_2021-12-07). Such batch can’t be processed by any version of Operate, as it has this gap in versions. This is the situation in which Operate can’t recover actually.

The only workaround that I can imagine is to play around with Elastic aliases for Zeebe indices. You could:

  1. Exclude all Zeebe indices of v. 1.2.x from aliases.
  2. Remove all Operate indices
  3. Start Operate of version 1.1.7 and wait till it imports all data (you could check index sizes, that they are not growing any more)
  4. Return Zeebe indices of v. 1.2.x to aliases.
  5. Start Operate of v. 1.2.4 - it will then process data of newer version

This is a hack of cause and should not be used in normal life. I see now that we must make it more clear in docs, that upgrade must either happen version by version, or before upgrade one must be sure that Operate processed all older Zeebe data.

Does this help you?

1 Like

Hi @svetlana

thank you for you analysis and suggestions.

Summary:
The 2-step-migration seemed to work without any noteworthy errors.
That said, when I create new instances or incidents, no figures are updated in Operate
nor any error visible in the logs. I guess, Operate does not see updates from Zeebe :man_shrugging: ?

What can I do to verify Operate does work?


Details:

I did as you proposed

  1. shut down zeebe cluster
  2. deleted the aliases from the 1.2.x indexes
    2.1. also deleted operate-* indexes
  3. started Operate 1.1.7 (it complained about missing connection to Zeebe)
  4. started Zeebe 1.2.5
  5. ingestion & migration started … without noticeable errors (except one I was not able to correlate, see below)
  6. stopped Operate
  7. re-added aliases to the 1.2.x indexes
  8. started Operate 1.2.6
  9. observed some warnings, but no errors.

The error I saw when Operate 1.1.7 was done with migration looks like (100+ occurences)

Dec 16 13:55:51 cluster-dev_blue-services-process-control-tower-operate-deployment-74f596955b-9wqpd process-control-tower-operate 2021-12-16 12:55:50.762  WARN 8 --- [nio-8080-exec-2] i.c.o.w.ForwardErrorController           : Requested non existing path /version. Forward (on serverside) to /"

I tend to ignore that … because it went away with 1.2.6


The full log of v1.2.6 looks like

INFO  i.c.o.Application                        : Starting Application using Java 11.0.13 on process-control-tower-operate-deployment-55d9bf9dfb-f78dw with PID 7 (/usr/local/operate/lib/operate-webapp-1.2.6-exec.jar started by root in /usr/local/operate)
INFO  i.c.o.Application                        : The following profiles are active: auth
INFO  trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.security.access.expression.method.DefaultMethodSecurityExpressionHandler@5528a42c' of type ...
INFO  trationDelegate$BeanPostProcessorChecker : Bean 'io.camunda.operate.webapp.security.CustomMethodSecurityExpressionHandler' of type ...
INFO  trationDelegate$BeanPostProcessorChecker : Bean 'io.camunda.operate.webapp.security.MethodSecurityConfig' of type ...
INFO  trationDelegate$BeanPostProcessorChecker : Bean 'methodSecurityMetadataSource' of type ...
INFO  o.s.b.w.e.t.TomcatWebServer              : Tomcat initialized with port(s): 8080 (http)
INFO  o.a.c.h.Http11NioProtocol                : Initializing ProtocolHandler ["http-nio-8080"]
INFO  o.a.c.c.StandardService                  : Starting service [Tomcat]
INFO  o.a.c.c.StandardEngine                   : Starting Servlet engine: [Apache Tomcat/9.0.53]
INFO  o.a.c.c.C.[.[.[/]                        : Initializing Spring embedded WebApplicationContext
INFO  w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 8770 ms
INFO  i.c.o.WebappModuleConfiguration          : Starting module: webapp
WARN  i.c.o.e.ElasticsearchConnector           : Username and/or password for are empty. Basic authentication for elasticsearch is not used.
WARN  i.c.o.e.ElasticsearchConnector           : Elasticsearch cluster is not accessible
INFO  i.c.o.s.SchemaStartup                    : SchemaStartup started.
INFO  i.c.o.s.SchemaStartup                    : SchemaStartup: validate schema.
INFO  i.c.o.s.SchemaStartup                    : SchemaStartup: schema is empty or not complete. Indices will be created.
INFO  i.c.o.s.ElasticsearchSchemaManager       : Create default settings from 'operate_template' with 1 shards and 0 replicas per index.
INFO  i.c.o.s.SchemaStartup                    : SchemaStartup: migrate schema.
INFO  i.c.o.s.m.ElasticsearchStepsRepository   : Read step 1.2.0-1_incident_script.json 
INFO  i.c.o.s.m.ElasticsearchStepsRepository   : Read step 1.2.0-3_user_script.json 
INFO  i.c.o.s.m.ElasticsearchStepsRepository   : Read step 1.2.0-4_user_script.json 
INFO  i.c.o.s.m.ElasticsearchStepsRepository   : Read step 1.2.0-2_user_script.json 
INFO  i.c.o.s.m.ElasticsearchStepsRepository   : Read step 1.2.0-5_user_delete-fields_script.json 
INFO  i.c.o.s.m.ElasticsearchStepsRepository   : Read step 1.2.0-0_list-view_script.json 
INFO  i.c.o.s.m.ElasticsearchStepsRepository   : Read step 1.1.0-0_variable_script.json 
INFO  i.c.o.s.m.ElasticsearchStepsRepository   : Add new step ProcessorStep [content={"script": {"lang": "painless","source": "if (ctx.joinRelation.name.equals('processInstance')) { ctx.treePath = 'PI_' + ctx.processInstanceKey; }"}}, appliedDate=null, indexName=list-view, isApplied=false, version=1.2.0, order=0, createdDate=2021-12-16T12:56:56.725657Z] to repository.
INFO  i.c.o.s.m.ElasticsearchStepsRepository   : Step ProcessorStep [content={"script": {"lang": "painless","source": "if (ctx.joinRelation.name.equals('processInstance')) { ctx.treePath = 'PI_' + ctx.processInstanceKey; }"}}, appliedDate=null, indexName=list-view, isApplied=false, version=1.2.0, order=0, createdDate=2021-12-16T12:56:56.725657Z]  saved.
INFO  i.c.o.s.m.ElasticsearchStepsRepository   : Add new step ProcessorStep [content={"script": {"lang": "painless","source": "ctx.treePath = 'PI_' + ctx.processInstanceKey + '/FN_' + ctx.flowNodeId + '/FNI_' + ctx.flowNodeInstanceKey;"}}, appliedDate=null, indexName=incident, isApplied=false, version=1.2.0, order=1, createdDate=2021-12-16T12:56:56.811913Z] to repository.
INFO  i.c.o.s.m.ElasticsearchStepsRepository   : Step ProcessorStep [content={"script": {"lang": "painless","source": "ctx.treePath = 'PI_' + ctx.processInstanceKey + '/FN_' + ctx.flowNodeId + '/FNI_' + ctx.flowNodeInstanceKey;"}}, appliedDate=null, indexName=incident, isApplied=false, version=1.2.0, order=1, createdDate=2021-12-16T12:56:56.811913Z]  saved.
INFO  i.c.o.s.m.ElasticsearchStepsRepository   : Add new step ProcessorStep [content={"remove": {"field": "role" }}, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=2, createdDate=2021-12-16T12:56:56.825655Z] to repository.
INFO  i.c.o.s.m.ElasticsearchStepsRepository   : Step ProcessorStep [content={"remove": {"field": "role" }}, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=2, createdDate=2021-12-16T12:56:56.825655Z]  saved.
INFO  i.c.o.s.m.ElasticsearchStepsRepository   : Add new step ProcessorStep [content={"append": {"field": "roles", "value": ["OWNER"]}}, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=3, createdDate=2021-12-16T12:56:56.838570Z] to repository.
INFO  i.c.o.s.m.ElasticsearchStepsRepository   : Step ProcessorStep [content={"append": {"field": "roles", "value": ["OWNER"]}}, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=3, createdDate=2021-12-16T12:56:56.838570Z]  saved.
INFO  i.c.o.s.m.ElasticsearchStepsRepository   : Add new step ProcessorStep [content={"script": {"lang": "painless","source": "ctx.userId = ctx.username; ctx.displayName = (ctx.firstname == null?'':ctx.firstname) + (ctx.lastname == null?'': (' ' + ctx.lastname))"}}, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=4, createdDate=2021-12-16T12:56:56.850682Z] to repository.
INFO  i.c.o.s.m.ElasticsearchStepsRepository   : Step ProcessorStep [content={"script": {"lang": "painless","source": "ctx.userId = ctx.username; ctx.displayName = (ctx.firstname == null?'':ctx.firstname) + (ctx.lastname == null?'': (' ' + ctx.lastname))"}}, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=4, createdDate=2021-12-16T12:56:56.850682Z]  saved.
INFO  i.c.o.s.m.ElasticsearchStepsRepository   : Add new step ProcessorStep [content={ "remove": { "field": ["username", "firstname", "lastname"] } }, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=5, createdDate=2021-12-16T12:56:56.863521Z] to repository.
INFO  i.c.o.s.m.ElasticsearchStepsRepository   : Step ProcessorStep [content={ "remove": { "field": ["username", "firstname", "lastname"] } }, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=5, createdDate=2021-12-16T12:56:56.863521Z]  saved.
INFO  i.c.o.s.m.Migrator                       : Check if index user needs to migrate.
INFO  i.c.o.s.m.Migrator                       : Check if index import-position needs to migrate.
INFO  i.c.o.s.m.Migrator                       : Check if index web-session needs to migrate.
INFO  i.c.o.s.m.Migrator                       : Check if index migration-steps-repository needs to migrate.
INFO  i.c.o.s.m.Migrator                       : Check if index process needs to migrate.
INFO  i.c.o.s.m.Migrator                       : No migration needed for import-position, no previous indices found.
INFO  i.c.o.s.m.Migrator                       : Check if index sequence-flow needs to migrate.
INFO  i.c.o.s.m.Migrator                       : No migration needed for sequence-flow, no previous indices found.
INFO  i.c.o.s.m.Migrator                       : Check if index incident needs to migrate.
INFO  i.c.o.s.m.Migrator                       : No migration needed for web-session, no previous indices found.
INFO  i.c.o.s.m.Migrator                       : Check if index batch-operation needs to migrate.
INFO  i.c.o.s.m.Migrator                       : No migration needed for migration-steps-repository, no previous indices found.
INFO  i.c.o.s.m.Migrator                       : Check if index variable needs to migrate.
INFO  i.c.o.s.m.Migrator                       : No migration needed for process, no previous indices found.
INFO  i.c.o.s.m.Migrator                       : Check if index event needs to migrate.
INFO  i.c.o.s.m.Migrator                       : No migration needed for variable, no previous indices found.
INFO  i.c.o.s.m.Migrator                       : Check if index flownode-instance needs to migrate.
INFO  i.c.o.s.m.Migrator                       : No migration needed for batch-operation, no previous indices found.
INFO  i.c.o.s.m.Migrator                       : No migration needed for event, no previous indices found.
INFO  i.c.o.s.m.Migrator                       : Check if index operation needs to migrate.
INFO  i.c.o.s.m.Migrator                       : Check if index list-view needs to migrate.
INFO  i.c.o.s.m.Migrator                       : No migration needed for operation, no previous indices found.
INFO  i.c.o.s.m.Migrator                       : No migration needed for flownode-instance, no previous indices found.
INFO  i.c.o.s.m.Migrator                       : Execute plan: ReindexPlan [steps=[ProcessorStep [content={"script": {"lang": "painless","source": "ctx.treePath = 'PI_' + ctx.processInstanceKey + '/FN_' + ctx.flowNodeId + '/FNI_' + ctx.flowNodeInstanceKey;"}}, appliedDate=null, indexName=incident, isApplied=false, version=1.2.0, order=1, createdDate=2021-12-16T12:56:56.811Z]],  srcIndex=operate-incident-1.0.0, dstIndex=operate-incident-1.2.0] 
INFO  i.c.o.s.m.Migrator                       : Execute plan: ReindexPlan [steps=[ProcessorStep [content={"script": {"lang": "painless","source": "if (ctx.joinRelation.name.equals('processInstance')) { ctx.treePath = 'PI_' + ctx.processInstanceKey; }"}}, appliedDate=null, indexName=list-view, isApplied=false, version=1.2.0, order=0, createdDate=2021-12-16T12:56:56.725Z]],  srcIndex=operate-list-view-1.1.0, dstIndex=operate-list-view-1.2.0] 
INFO  i.c.o.s.m.Migrator                       : Execute plan: ReindexPlan [steps=[ProcessorStep [content={"remove": {"field": "role" }}, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=2, createdDate=2021-12-16T12:56:56.825Z], ProcessorStep [content={"append": {"field": "roles", "value": ["OWNER"]}}, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=3, createdDate=2021-12-16T12:56:56.838Z], ProcessorStep [content={"script": {"lang": "painless","source": "ctx.userId = ctx.username; ctx.displayName = (ctx.firstname == null?'':ctx.firstname) + (ctx.lastname == null?'': (' ' + ctx.lastname))"}}, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=4, createdDate=2021-12-16T12:56:56.850Z], ProcessorStep [content={ "remove": { "field": ["username", "firstname", "lastname"] } }, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=5, createdDate=2021-12-16T12:56:56.863Z]],  srcIndex=operate-user-1.0.0, dstIndex=operate-user-1.2.0] 
INFO  i.c.o.e.RetryElasticsearchClient         : Source docs: 37, Migrated docs: 37
INFO  i.c.o.e.RetryElasticsearchClient         : Source docs: 2, Migrated docs: 2
INFO  i.c.o.s.m.ElasticsearchStepsRepository   : Step ProcessorStep [content={"script": {"lang": "painless","source": "ctx.treePath = 'PI_' + ctx.processInstanceKey + '/FN_' + ctx.flowNodeId + '/FNI_' + ctx.flowNodeInstanceKey;"}}, appliedDate=2021-12-16T12:57:00.543907Z, indexName=incident, isApplied=true, version=1.2.0, order=1, createdDate=2021-12-16T12:56:56.811Z]  saved.
INFO  i.c.o.s.m.ElasticsearchStepsRepository   : Step ProcessorStep [content={"remove": {"field": "role" }}, appliedDate=2021-12-16T12:57:00.589047Z, indexName=user, isApplied=true, version=1.2.0, order=2, createdDate=2021-12-16T12:56:56.825Z]  saved.
INFO  i.c.o.s.m.ElasticsearchStepsRepository   : Step ProcessorStep [content={"append": {"field": "roles", "value": ["OWNER"]}}, appliedDate=2021-12-16T12:57:00.603771Z, indexName=user, isApplied=true, version=1.2.0, order=3, createdDate=2021-12-16T12:56:56.838Z]  saved.
INFO  i.c.o.s.m.ElasticsearchStepsRepository   : Step ProcessorStep [content={"script": {"lang": "painless","source": "ctx.userId = ctx.username; ctx.displayName = (ctx.firstname == null?'':ctx.firstname) + (ctx.lastname == null?'': (' ' + ctx.lastname))"}}, appliedDate=2021-12-16T12:57:00.618469Z, indexName=user, isApplied=true, version=1.2.0, order=4, createdDate=2021-12-16T12:56:56.850Z]  saved.
INFO  i.c.o.s.m.ElasticsearchStepsRepository   : Step ProcessorStep [content={ "remove": { "field": ["username", "firstname", "lastname"] } }, appliedDate=2021-12-16T12:57:00.633431Z, indexName=user, isApplied=true, version=1.2.0, order=5, createdDate=2021-12-16T12:56:56.863Z]  saved.
INFO  i.c.o.s.m.Migrator                       : Refresh index operate-incident-1.2.0_*
INFO  i.c.o.s.m.Migrator                       : Deleted previous indices for pattern operate-incident-1.0.0_*
INFO  i.c.o.s.m.Migrator                       : Refresh index operate-user-1.2.0_*
INFO  i.c.o.s.m.Migrator                       : Deleted previous indices for pattern operate-user-1.0.0_*
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
INFO  i.c.o.e.RetryElasticsearchClient         : Source docs: 82879, Migrated docs: 82879
INFO  i.c.o.s.m.ElasticsearchStepsRepository   : Step ProcessorStep [content={"script": {"lang": "painless","source": "if (ctx.joinRelation.name.equals('processInstance')) { ctx.treePath = 'PI_' + ctx.processInstanceKey; }"}}, appliedDate=2021-12-16T12:57:26.124411Z, indexName=list-view, isApplied=true, version=1.2.0, order=0, createdDate=2021-12-16T12:56:56.725Z]  saved.
INFO  i.c.o.s.m.Migrator                       : Refresh index operate-list-view-1.2.0_*
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
INFO  i.c.o.s.m.Migrator                       : Deleted previous indices for pattern operate-list-view-1.1.0_*
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-list-view-1.1.0_2021-11-26/m9XfPV2BTy-4rak1kCxbfg], [operate-list-view-1.1.0_2021-11-08/5tZ9-abFTB6B_IiTWYEXcw], [operate-list-view-1.1.0_2021-11-15/FIi5Z6bBSUGUsbmmpHiwYw], [operate-list-view-1.1.0_/zhZ7jxCMRomFXkruul_C9g], [operate-list-view-1.1.0_2021-11-11/3aBIdSRiRJuh6XLGRLk5tQ], [operate-list-view-1.1.0_2021-11-16/GrFe8U8bSIO6IdkbhLlTrw], [operate-list-view-1.1.0_2021-12-01/gHYSF3ZCR7yPHCa7snhU-g], [operate-list-view-1.1.0_2021-11-12/rotHJN9lSLuhb0ODBAhfGA], [operate-list-view-1.1.0_2021-11-17/U8uvaN0aQJ6ZVAJxMxKvww], [operate-list-view-1.1.0_2021-11-30/GEm28FwZRX-ZfYfOEA-Ziw]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-list-view-1.1.0_2021-11-26/m9XfPV2BTy-4rak1kCxbfg], [operate-list-view-1.1.0_2021-11-08/5tZ9-abFTB6B_IiTWYEXcw], [operate-list-view-1.1.0_2021-11-15/FIi5Z6bBSUGUsbmmpHiwYw], [operate-list-view-1.1.0_/zhZ7jxCMRomFXkruul_C9g], [operate-list-view-1.1.0_2021-11-11/3aBIdSRiRJuh6XLGRLk5tQ], [operate-list-view-1.1.0_2021-11-16/GrFe8U8bSIO6IdkbhLlTrw], [operate-list-view-1.1.0_2021-12-01/gHYSF3ZCR7yPHCa7snhU-g], [operate-list-view-1.1.0_2021-11-12/rotHJN9lSLuhb0ODBAhfGA], [operate-list-view-1.1.0_2021-11-17/U8uvaN0aQJ6ZVAJxMxKvww], [operate-list-view-1.1.0_2021-11-30/GEm28FwZRX-ZfYfOEA-Ziw]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-list-view-1.1.0_2021-11-26/m9XfPV2BTy-4rak1kCxbfg], [operate-list-view-1.1.0_2021-11-08/5tZ9-abFTB6B_IiTWYEXcw], [operate-list-view-1.1.0_2021-11-15/FIi5Z6bBSUGUsbmmpHiwYw], [operate-list-view-1.1.0_/zhZ7jxCMRomFXkruul_C9g], [operate-list-view-1.1.0_2021-11-11/3aBIdSRiRJuh6XLGRLk5tQ], [operate-list-view-1.1.0_2021-11-16/GrFe8U8bSIO6IdkbhLlTrw], [operate-list-view-1.1.0_2021-12-01/gHYSF3ZCR7yPHCa7snhU-g], [operate-list-view-1.1.0_2021-11-12/rotHJN9lSLuhb0ODBAhfGA], [operate-list-view-1.1.0_2021-11-17/U8uvaN0aQJ6ZVAJxMxKvww], [operate-list-view-1.1.0_2021-11-30/GEm28FwZRX-ZfYfOEA-Ziw]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN  i.c.o.u.RetryOperation                   : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-list-view-1.1.0_2021-11-26/m9XfPV2BTy-4rak1kCxbfg], [operate-list-view-1.1.0_2021-11-08/5tZ9-abFTB6B_IiTWYEXcw], [operate-list-view-1.1.0_2021-11-15/FIi5Z6bBSUGUsbmmpHiwYw], [operate-list-view-1.1.0_/zhZ7jxCMRomFXkruul_C9g], [operate-list-view-1.1.0_2021-11-11/3aBIdSRiRJuh6XLGRLk5tQ], [operate-list-view-1.1.0_2021-11-16/GrFe8U8bSIO6IdkbhLlTrw], [operate-list-view-1.1.0_2021-12-01/gHYSF3ZCR7yPHCa7snhU-g], [operate-list-view-1.1.0_2021-11-12/rotHJN9lSLuhb0ODBAhfGA], [operate-list-view-1.1.0_2021-11-17/U8uvaN0aQJ6ZVAJxMxKvww], [operate-list-view-1.1.0_2021-11-30/GEm28FwZRX-ZfYfOEA-Ziw]]. Try again after snapshot finishes or cancel the currently running snapshot.]
INFO  i.c.o.s.m.Migrator                       : Deleted previous templates for operate-incident-1.0.0_template
INFO  i.c.o.s.m.Migrator                       : Deleted previous templates for operate-list-view-1.1.0_template
INFO  i.c.o.s.SchemaStartup                    : SchemaStartup finished.
INFO  i.c.o.ImportModuleConfiguration          : Starting module: importer
INFO  i.c.o.ArchiverModuleConfiguration        : Starting module: archiver
WARN  i.c.o.e.ElasticsearchConnector           : Username and/or password for are empty. Basic authentication for elasticsearch is not used.
WARN  i.c.o.e.ElasticsearchConnector           : Elasticsearch cluster is not accessible
INFO  i.c.o.w.StartupBean                      : INIT: Create users in elasticsearch if not exists ...
INFO  .o.w.s.e.ElasticSearchUserDetailsService : Create user in ElasticSearch for userId demo
INFO  .o.w.s.e.ElasticSearchUserDetailsService : Create user in ElasticSearch for userId view
INFO  i.c.o.w.StartupBean                      : INIT: Start operation executor...
INFO  i.c.o.w.StartupBean                      : INIT: DONE
INFO  i.c.o.z.ZeebeImporter                    : INIT: Start importing data...
INFO  i.c.o.a.Archiver                         : INIT: Start archiving data...
INFO  i.c.o.a.Archiver                         : Starting archiver for partitions: [1, 2, 3]
INFO  i.c.o.z.RecordsReaderHolder              : Starting import for partitions: [1, 2, 3]
INFO  o.s.b.a.e.w.EndpointLinksResolver        : Exposing 2 endpoint(s) beneath base path '/actuator'
INFO  o.s.s.w.DefaultSecurityFilterChain       : Will secure any request with ...
INFO  o.h.v.i.u.Version                        : HV000001: Hibernate Validator 6.2.0.Final
INFO  o.s.b.a.w.s.WelcomePageHandlerMapping    : Adding welcome page: class path resource [META-INF/resources/index.html]
INFO  o.s.l.c.s.AbstractContextSource          : Property 'userDn' not set - anonymous context will be used for read-write operations
INFO  o.a.c.h.Http11NioProtocol                : Starting ProtocolHandler ["http-nio-8080"]
INFO  o.s.b.w.e.t.TomcatWebServer              : Tomcat started on port(s): 8080 (http) with context path ''
INFO  i.c.o.Application                        : Started Application in 67.087 seconds (JVM running for 71.62)
INFO  i.c.o.s.m.SchemaMigration                : SchemaMigration finished.
INFO  o.a.c.c.C.[.[.[/]                        : Initializing Spring DispatcherServlet 'dispatcherServlet'
INFO  o.s.w.s.DispatcherServlet                : Initializing Servlet 'dispatcherServlet'
INFO  o.s.w.s.DispatcherServlet                : Completed initialization in 2 ms