Hi @svetlana
thank you for you analysis and suggestions.
Summary:
The 2-step-migration seemed to work without any noteworthy errors.
That said, when I create new instances or incidents, no figures are updated in Operate
nor any error visible in the logs. I guess, Operate does not see updates from Zeebe ?
What can I do to verify Operate does work?
Details:
I did as you proposed
- shut down zeebe cluster
- deleted the aliases from the 1.2.x indexes
2.1. also deleted operate-* indexes
- started Operate 1.1.7 (it complained about missing connection to Zeebe)
- started Zeebe 1.2.5
- ingestion & migration started … without noticeable errors (except one I was not able to correlate, see below)
- stopped Operate
- re-added aliases to the 1.2.x indexes
- started Operate 1.2.6
- observed some warnings, but no errors.
The error I saw when Operate 1.1.7 was done with migration looks like (100+ occurences)
Dec 16 13:55:51 cluster-dev_blue-services-process-control-tower-operate-deployment-74f596955b-9wqpd process-control-tower-operate 2021-12-16 12:55:50.762 WARN 8 --- [nio-8080-exec-2] i.c.o.w.ForwardErrorController : Requested non existing path /version. Forward (on serverside) to /"
I tend to ignore that … because it went away with 1.2.6
The full log of v1.2.6 looks like
INFO i.c.o.Application : Starting Application using Java 11.0.13 on process-control-tower-operate-deployment-55d9bf9dfb-f78dw with PID 7 (/usr/local/operate/lib/operate-webapp-1.2.6-exec.jar started by root in /usr/local/operate)
INFO i.c.o.Application : The following profiles are active: auth
INFO trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.security.access.expression.method.DefaultMethodSecurityExpressionHandler@5528a42c' of type ...
INFO trationDelegate$BeanPostProcessorChecker : Bean 'io.camunda.operate.webapp.security.CustomMethodSecurityExpressionHandler' of type ...
INFO trationDelegate$BeanPostProcessorChecker : Bean 'io.camunda.operate.webapp.security.MethodSecurityConfig' of type ...
INFO trationDelegate$BeanPostProcessorChecker : Bean 'methodSecurityMetadataSource' of type ...
INFO o.s.b.w.e.t.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
INFO o.a.c.h.Http11NioProtocol : Initializing ProtocolHandler ["http-nio-8080"]
INFO o.a.c.c.StandardService : Starting service [Tomcat]
INFO o.a.c.c.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.53]
INFO o.a.c.c.C.[.[.[/] : Initializing Spring embedded WebApplicationContext
INFO w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 8770 ms
INFO i.c.o.WebappModuleConfiguration : Starting module: webapp
WARN i.c.o.e.ElasticsearchConnector : Username and/or password for are empty. Basic authentication for elasticsearch is not used.
WARN i.c.o.e.ElasticsearchConnector : Elasticsearch cluster is not accessible
INFO i.c.o.s.SchemaStartup : SchemaStartup started.
INFO i.c.o.s.SchemaStartup : SchemaStartup: validate schema.
INFO i.c.o.s.SchemaStartup : SchemaStartup: schema is empty or not complete. Indices will be created.
INFO i.c.o.s.ElasticsearchSchemaManager : Create default settings from 'operate_template' with 1 shards and 0 replicas per index.
INFO i.c.o.s.SchemaStartup : SchemaStartup: migrate schema.
INFO i.c.o.s.m.ElasticsearchStepsRepository : Read step 1.2.0-1_incident_script.json
INFO i.c.o.s.m.ElasticsearchStepsRepository : Read step 1.2.0-3_user_script.json
INFO i.c.o.s.m.ElasticsearchStepsRepository : Read step 1.2.0-4_user_script.json
INFO i.c.o.s.m.ElasticsearchStepsRepository : Read step 1.2.0-2_user_script.json
INFO i.c.o.s.m.ElasticsearchStepsRepository : Read step 1.2.0-5_user_delete-fields_script.json
INFO i.c.o.s.m.ElasticsearchStepsRepository : Read step 1.2.0-0_list-view_script.json
INFO i.c.o.s.m.ElasticsearchStepsRepository : Read step 1.1.0-0_variable_script.json
INFO i.c.o.s.m.ElasticsearchStepsRepository : Add new step ProcessorStep [content={"script": {"lang": "painless","source": "if (ctx.joinRelation.name.equals('processInstance')) { ctx.treePath = 'PI_' + ctx.processInstanceKey; }"}}, appliedDate=null, indexName=list-view, isApplied=false, version=1.2.0, order=0, createdDate=2021-12-16T12:56:56.725657Z] to repository.
INFO i.c.o.s.m.ElasticsearchStepsRepository : Step ProcessorStep [content={"script": {"lang": "painless","source": "if (ctx.joinRelation.name.equals('processInstance')) { ctx.treePath = 'PI_' + ctx.processInstanceKey; }"}}, appliedDate=null, indexName=list-view, isApplied=false, version=1.2.0, order=0, createdDate=2021-12-16T12:56:56.725657Z] saved.
INFO i.c.o.s.m.ElasticsearchStepsRepository : Add new step ProcessorStep [content={"script": {"lang": "painless","source": "ctx.treePath = 'PI_' + ctx.processInstanceKey + '/FN_' + ctx.flowNodeId + '/FNI_' + ctx.flowNodeInstanceKey;"}}, appliedDate=null, indexName=incident, isApplied=false, version=1.2.0, order=1, createdDate=2021-12-16T12:56:56.811913Z] to repository.
INFO i.c.o.s.m.ElasticsearchStepsRepository : Step ProcessorStep [content={"script": {"lang": "painless","source": "ctx.treePath = 'PI_' + ctx.processInstanceKey + '/FN_' + ctx.flowNodeId + '/FNI_' + ctx.flowNodeInstanceKey;"}}, appliedDate=null, indexName=incident, isApplied=false, version=1.2.0, order=1, createdDate=2021-12-16T12:56:56.811913Z] saved.
INFO i.c.o.s.m.ElasticsearchStepsRepository : Add new step ProcessorStep [content={"remove": {"field": "role" }}, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=2, createdDate=2021-12-16T12:56:56.825655Z] to repository.
INFO i.c.o.s.m.ElasticsearchStepsRepository : Step ProcessorStep [content={"remove": {"field": "role" }}, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=2, createdDate=2021-12-16T12:56:56.825655Z] saved.
INFO i.c.o.s.m.ElasticsearchStepsRepository : Add new step ProcessorStep [content={"append": {"field": "roles", "value": ["OWNER"]}}, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=3, createdDate=2021-12-16T12:56:56.838570Z] to repository.
INFO i.c.o.s.m.ElasticsearchStepsRepository : Step ProcessorStep [content={"append": {"field": "roles", "value": ["OWNER"]}}, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=3, createdDate=2021-12-16T12:56:56.838570Z] saved.
INFO i.c.o.s.m.ElasticsearchStepsRepository : Add new step ProcessorStep [content={"script": {"lang": "painless","source": "ctx.userId = ctx.username; ctx.displayName = (ctx.firstname == null?'':ctx.firstname) + (ctx.lastname == null?'': (' ' + ctx.lastname))"}}, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=4, createdDate=2021-12-16T12:56:56.850682Z] to repository.
INFO i.c.o.s.m.ElasticsearchStepsRepository : Step ProcessorStep [content={"script": {"lang": "painless","source": "ctx.userId = ctx.username; ctx.displayName = (ctx.firstname == null?'':ctx.firstname) + (ctx.lastname == null?'': (' ' + ctx.lastname))"}}, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=4, createdDate=2021-12-16T12:56:56.850682Z] saved.
INFO i.c.o.s.m.ElasticsearchStepsRepository : Add new step ProcessorStep [content={ "remove": { "field": ["username", "firstname", "lastname"] } }, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=5, createdDate=2021-12-16T12:56:56.863521Z] to repository.
INFO i.c.o.s.m.ElasticsearchStepsRepository : Step ProcessorStep [content={ "remove": { "field": ["username", "firstname", "lastname"] } }, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=5, createdDate=2021-12-16T12:56:56.863521Z] saved.
INFO i.c.o.s.m.Migrator : Check if index user needs to migrate.
INFO i.c.o.s.m.Migrator : Check if index import-position needs to migrate.
INFO i.c.o.s.m.Migrator : Check if index web-session needs to migrate.
INFO i.c.o.s.m.Migrator : Check if index migration-steps-repository needs to migrate.
INFO i.c.o.s.m.Migrator : Check if index process needs to migrate.
INFO i.c.o.s.m.Migrator : No migration needed for import-position, no previous indices found.
INFO i.c.o.s.m.Migrator : Check if index sequence-flow needs to migrate.
INFO i.c.o.s.m.Migrator : No migration needed for sequence-flow, no previous indices found.
INFO i.c.o.s.m.Migrator : Check if index incident needs to migrate.
INFO i.c.o.s.m.Migrator : No migration needed for web-session, no previous indices found.
INFO i.c.o.s.m.Migrator : Check if index batch-operation needs to migrate.
INFO i.c.o.s.m.Migrator : No migration needed for migration-steps-repository, no previous indices found.
INFO i.c.o.s.m.Migrator : Check if index variable needs to migrate.
INFO i.c.o.s.m.Migrator : No migration needed for process, no previous indices found.
INFO i.c.o.s.m.Migrator : Check if index event needs to migrate.
INFO i.c.o.s.m.Migrator : No migration needed for variable, no previous indices found.
INFO i.c.o.s.m.Migrator : Check if index flownode-instance needs to migrate.
INFO i.c.o.s.m.Migrator : No migration needed for batch-operation, no previous indices found.
INFO i.c.o.s.m.Migrator : No migration needed for event, no previous indices found.
INFO i.c.o.s.m.Migrator : Check if index operation needs to migrate.
INFO i.c.o.s.m.Migrator : Check if index list-view needs to migrate.
INFO i.c.o.s.m.Migrator : No migration needed for operation, no previous indices found.
INFO i.c.o.s.m.Migrator : No migration needed for flownode-instance, no previous indices found.
INFO i.c.o.s.m.Migrator : Execute plan: ReindexPlan [steps=[ProcessorStep [content={"script": {"lang": "painless","source": "ctx.treePath = 'PI_' + ctx.processInstanceKey + '/FN_' + ctx.flowNodeId + '/FNI_' + ctx.flowNodeInstanceKey;"}}, appliedDate=null, indexName=incident, isApplied=false, version=1.2.0, order=1, createdDate=2021-12-16T12:56:56.811Z]], srcIndex=operate-incident-1.0.0, dstIndex=operate-incident-1.2.0]
INFO i.c.o.s.m.Migrator : Execute plan: ReindexPlan [steps=[ProcessorStep [content={"script": {"lang": "painless","source": "if (ctx.joinRelation.name.equals('processInstance')) { ctx.treePath = 'PI_' + ctx.processInstanceKey; }"}}, appliedDate=null, indexName=list-view, isApplied=false, version=1.2.0, order=0, createdDate=2021-12-16T12:56:56.725Z]], srcIndex=operate-list-view-1.1.0, dstIndex=operate-list-view-1.2.0]
INFO i.c.o.s.m.Migrator : Execute plan: ReindexPlan [steps=[ProcessorStep [content={"remove": {"field": "role" }}, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=2, createdDate=2021-12-16T12:56:56.825Z], ProcessorStep [content={"append": {"field": "roles", "value": ["OWNER"]}}, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=3, createdDate=2021-12-16T12:56:56.838Z], ProcessorStep [content={"script": {"lang": "painless","source": "ctx.userId = ctx.username; ctx.displayName = (ctx.firstname == null?'':ctx.firstname) + (ctx.lastname == null?'': (' ' + ctx.lastname))"}}, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=4, createdDate=2021-12-16T12:56:56.850Z], ProcessorStep [content={ "remove": { "field": ["username", "firstname", "lastname"] } }, appliedDate=null, indexName=user, isApplied=false, version=1.2.0, order=5, createdDate=2021-12-16T12:56:56.863Z]], srcIndex=operate-user-1.0.0, dstIndex=operate-user-1.2.0]
INFO i.c.o.e.RetryElasticsearchClient : Source docs: 37, Migrated docs: 37
INFO i.c.o.e.RetryElasticsearchClient : Source docs: 2, Migrated docs: 2
INFO i.c.o.s.m.ElasticsearchStepsRepository : Step ProcessorStep [content={"script": {"lang": "painless","source": "ctx.treePath = 'PI_' + ctx.processInstanceKey + '/FN_' + ctx.flowNodeId + '/FNI_' + ctx.flowNodeInstanceKey;"}}, appliedDate=2021-12-16T12:57:00.543907Z, indexName=incident, isApplied=true, version=1.2.0, order=1, createdDate=2021-12-16T12:56:56.811Z] saved.
INFO i.c.o.s.m.ElasticsearchStepsRepository : Step ProcessorStep [content={"remove": {"field": "role" }}, appliedDate=2021-12-16T12:57:00.589047Z, indexName=user, isApplied=true, version=1.2.0, order=2, createdDate=2021-12-16T12:56:56.825Z] saved.
INFO i.c.o.s.m.ElasticsearchStepsRepository : Step ProcessorStep [content={"append": {"field": "roles", "value": ["OWNER"]}}, appliedDate=2021-12-16T12:57:00.603771Z, indexName=user, isApplied=true, version=1.2.0, order=3, createdDate=2021-12-16T12:56:56.838Z] saved.
INFO i.c.o.s.m.ElasticsearchStepsRepository : Step ProcessorStep [content={"script": {"lang": "painless","source": "ctx.userId = ctx.username; ctx.displayName = (ctx.firstname == null?'':ctx.firstname) + (ctx.lastname == null?'': (' ' + ctx.lastname))"}}, appliedDate=2021-12-16T12:57:00.618469Z, indexName=user, isApplied=true, version=1.2.0, order=4, createdDate=2021-12-16T12:56:56.850Z] saved.
INFO i.c.o.s.m.ElasticsearchStepsRepository : Step ProcessorStep [content={ "remove": { "field": ["username", "firstname", "lastname"] } }, appliedDate=2021-12-16T12:57:00.633431Z, indexName=user, isApplied=true, version=1.2.0, order=5, createdDate=2021-12-16T12:56:56.863Z] saved.
INFO i.c.o.s.m.Migrator : Refresh index operate-incident-1.2.0_*
INFO i.c.o.s.m.Migrator : Deleted previous indices for pattern operate-incident-1.0.0_*
INFO i.c.o.s.m.Migrator : Refresh index operate-user-1.2.0_*
INFO i.c.o.s.m.Migrator : Deleted previous indices for pattern operate-user-1.0.0_*
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
INFO i.c.o.e.RetryElasticsearchClient : Source docs: 82879, Migrated docs: 82879
INFO i.c.o.s.m.ElasticsearchStepsRepository : Step ProcessorStep [content={"script": {"lang": "painless","source": "if (ctx.joinRelation.name.equals('processInstance')) { ctx.treePath = 'PI_' + ctx.processInstanceKey; }"}}, appliedDate=2021-12-16T12:57:26.124411Z, indexName=list-view, isApplied=true, version=1.2.0, order=0, createdDate=2021-12-16T12:56:56.725Z] saved.
INFO i.c.o.s.m.Migrator : Refresh index operate-list-view-1.2.0_*
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
INFO i.c.o.s.m.Migrator : Deleted previous indices for pattern operate-list-view-1.1.0_*
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-list-view-1.1.0_2021-11-26/m9XfPV2BTy-4rak1kCxbfg], [operate-list-view-1.1.0_2021-11-08/5tZ9-abFTB6B_IiTWYEXcw], [operate-list-view-1.1.0_2021-11-15/FIi5Z6bBSUGUsbmmpHiwYw], [operate-list-view-1.1.0_/zhZ7jxCMRomFXkruul_C9g], [operate-list-view-1.1.0_2021-11-11/3aBIdSRiRJuh6XLGRLk5tQ], [operate-list-view-1.1.0_2021-11-16/GrFe8U8bSIO6IdkbhLlTrw], [operate-list-view-1.1.0_2021-12-01/gHYSF3ZCR7yPHCa7snhU-g], [operate-list-view-1.1.0_2021-11-12/rotHJN9lSLuhb0ODBAhfGA], [operate-list-view-1.1.0_2021-11-17/U8uvaN0aQJ6ZVAJxMxKvww], [operate-list-view-1.1.0_2021-11-30/GEm28FwZRX-ZfYfOEA-Ziw]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-list-view-1.1.0_2021-11-26/m9XfPV2BTy-4rak1kCxbfg], [operate-list-view-1.1.0_2021-11-08/5tZ9-abFTB6B_IiTWYEXcw], [operate-list-view-1.1.0_2021-11-15/FIi5Z6bBSUGUsbmmpHiwYw], [operate-list-view-1.1.0_/zhZ7jxCMRomFXkruul_C9g], [operate-list-view-1.1.0_2021-11-11/3aBIdSRiRJuh6XLGRLk5tQ], [operate-list-view-1.1.0_2021-11-16/GrFe8U8bSIO6IdkbhLlTrw], [operate-list-view-1.1.0_2021-12-01/gHYSF3ZCR7yPHCa7snhU-g], [operate-list-view-1.1.0_2021-11-12/rotHJN9lSLuhb0ODBAhfGA], [operate-list-view-1.1.0_2021-11-17/U8uvaN0aQJ6ZVAJxMxKvww], [operate-list-view-1.1.0_2021-11-30/GEm28FwZRX-ZfYfOEA-Ziw]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-list-view-1.1.0_2021-11-26/m9XfPV2BTy-4rak1kCxbfg], [operate-list-view-1.1.0_2021-11-08/5tZ9-abFTB6B_IiTWYEXcw], [operate-list-view-1.1.0_2021-11-15/FIi5Z6bBSUGUsbmmpHiwYw], [operate-list-view-1.1.0_/zhZ7jxCMRomFXkruul_C9g], [operate-list-view-1.1.0_2021-11-11/3aBIdSRiRJuh6XLGRLk5tQ], [operate-list-view-1.1.0_2021-11-16/GrFe8U8bSIO6IdkbhLlTrw], [operate-list-view-1.1.0_2021-12-01/gHYSF3ZCR7yPHCa7snhU-g], [operate-list-view-1.1.0_2021-11-12/rotHJN9lSLuhb0ODBAhfGA], [operate-list-view-1.1.0_2021-11-17/U8uvaN0aQJ6ZVAJxMxKvww], [operate-list-view-1.1.0_2021-11-30/GEm28FwZRX-ZfYfOEA-Ziw]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-incident-1.0.0_/4bkvZfNEQBCbgYx8wUMJ7g]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-user-1.0.0_/MamCpzWoRrOdi9VsTNjVGg]]. Try again after snapshot finishes or cancel the currently running snapshot.]
WARN i.c.o.u.RetryOperation : Elasticsearch exception [type=snapshot_in_progress_exception, reason=Cannot delete indices that are being snapshotted: [[operate-list-view-1.1.0_2021-11-26/m9XfPV2BTy-4rak1kCxbfg], [operate-list-view-1.1.0_2021-11-08/5tZ9-abFTB6B_IiTWYEXcw], [operate-list-view-1.1.0_2021-11-15/FIi5Z6bBSUGUsbmmpHiwYw], [operate-list-view-1.1.0_/zhZ7jxCMRomFXkruul_C9g], [operate-list-view-1.1.0_2021-11-11/3aBIdSRiRJuh6XLGRLk5tQ], [operate-list-view-1.1.0_2021-11-16/GrFe8U8bSIO6IdkbhLlTrw], [operate-list-view-1.1.0_2021-12-01/gHYSF3ZCR7yPHCa7snhU-g], [operate-list-view-1.1.0_2021-11-12/rotHJN9lSLuhb0ODBAhfGA], [operate-list-view-1.1.0_2021-11-17/U8uvaN0aQJ6ZVAJxMxKvww], [operate-list-view-1.1.0_2021-11-30/GEm28FwZRX-ZfYfOEA-Ziw]]. Try again after snapshot finishes or cancel the currently running snapshot.]
INFO i.c.o.s.m.Migrator : Deleted previous templates for operate-incident-1.0.0_template
INFO i.c.o.s.m.Migrator : Deleted previous templates for operate-list-view-1.1.0_template
INFO i.c.o.s.SchemaStartup : SchemaStartup finished.
INFO i.c.o.ImportModuleConfiguration : Starting module: importer
INFO i.c.o.ArchiverModuleConfiguration : Starting module: archiver
WARN i.c.o.e.ElasticsearchConnector : Username and/or password for are empty. Basic authentication for elasticsearch is not used.
WARN i.c.o.e.ElasticsearchConnector : Elasticsearch cluster is not accessible
INFO i.c.o.w.StartupBean : INIT: Create users in elasticsearch if not exists ...
INFO .o.w.s.e.ElasticSearchUserDetailsService : Create user in ElasticSearch for userId demo
INFO .o.w.s.e.ElasticSearchUserDetailsService : Create user in ElasticSearch for userId view
INFO i.c.o.w.StartupBean : INIT: Start operation executor...
INFO i.c.o.w.StartupBean : INIT: DONE
INFO i.c.o.z.ZeebeImporter : INIT: Start importing data...
INFO i.c.o.a.Archiver : INIT: Start archiving data...
INFO i.c.o.a.Archiver : Starting archiver for partitions: [1, 2, 3]
INFO i.c.o.z.RecordsReaderHolder : Starting import for partitions: [1, 2, 3]
INFO o.s.b.a.e.w.EndpointLinksResolver : Exposing 2 endpoint(s) beneath base path '/actuator'
INFO o.s.s.w.DefaultSecurityFilterChain : Will secure any request with ...
INFO o.h.v.i.u.Version : HV000001: Hibernate Validator 6.2.0.Final
INFO o.s.b.a.w.s.WelcomePageHandlerMapping : Adding welcome page: class path resource [META-INF/resources/index.html]
INFO o.s.l.c.s.AbstractContextSource : Property 'userDn' not set - anonymous context will be used for read-write operations
INFO o.a.c.h.Http11NioProtocol : Starting ProtocolHandler ["http-nio-8080"]
INFO o.s.b.w.e.t.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
INFO i.c.o.Application : Started Application in 67.087 seconds (JVM running for 71.62)
INFO i.c.o.s.m.SchemaMigration : SchemaMigration finished.
INFO o.a.c.c.C.[.[.[/] : Initializing Spring DispatcherServlet 'dispatcherServlet'
INFO o.s.w.s.DispatcherServlet : Initializing Servlet 'dispatcherServlet'
INFO o.s.w.s.DispatcherServlet : Completed initialization in 2 ms