Operate exeeds search.max_open_scroll_context on AWS using Opensearch

We are running camunda 10.0.4 via EKS, using an aws managed Opensearch 2.11.1 cluster, with 2 nodes, and have issues with Operate, that seems to saturate the scroll context of Opensearch.

All was running smooth, until somewhere last week a developer was running a test process, and did a bulk cancel via operate. Since then, the operate user interface is misbehaving. For a few days now, we see issues where Operate is overloading the Opensearch cluster (by exceeding the max_open_scroll_context=500 limit continuously). It sometimes shows process instance info, but more often not (then it helps refreshing 5 or 6 times until it suddenly shows something again). In operate, still there are about 100 instances shown to be still canceling (for 3 days now), that seem to be stuck now.

Increasing the limit of 500 open scroll contexts via AWS is not an option, since this limit is a hard enforced limit, managed by aws, not allowed to be changed in the cluster.

The exceeding of max_open_scroll_context can be seen in the /_nodes/stats/indices/search call, where both nodes have a “scroll_current”:500 ;

{"_nodes":{"total":2,"successful":2,"failed":0},"cluster_name":"opensearch","nodes":{"1Tdt2aBnSPuUA_n5XZQytg":{"timestamp":1718103102779,"name":"f4017cda3462a810ee8d3ee7b0845ab9","roles":["data","ingest","master","remote_cluster_client"],"indices":{"search":{"open_contexts":500,"query_total":30533548,"query_time_in_millis":5039314,"query_current":0,"fetch_total":1467197,"fetch_time_in_millis":676315,"fetch_current":0,"scroll_total":2520280,"scroll_time_in_millis":231961399070,"scroll_current":500,"point_in_time_total":0,"point_in_time_time_in_millis":0,"point_in_time_current":0,"suggest_total":0,"suggest_time_in_millis":0,"suggest_current":0,"request":{"dfs_pre_query":{"time_in_millis":0,"current":0,"total":0},"query":{"time_in_millis":18982857,"current":924843,"total":1555161},"fetch":{"time_in_millis":1055078,"current":0,"total":1555161},"dfs_query":{"time_in_millis":0,"current":0,"total":0},"expand":{"time_in_millis":5863815,"current":0,"total":1555161},"can_match":{"time_in_millis":6064734,"current":0,"total":924843}}}}},"GWDOGkNcT5mbDqrjatpDYw":{"timestamp":1718103102780,"name":"c69b1b5babd14f1aee46f5ca1f43faf6","roles":["data","ingest","master","remote_cluster_client"],"indices":{"search":{"open_contexts":500,"query_total":33557847,"query_time_in_millis":5325534,"query_current":0,"fetch_total":2183208,"fetch_time_in_millis":5861040,"fetch_current":0,"scroll_total":2556575,"scroll_time_in_millis":234506794018,"scroll_current":500,"point_in_time_total":0,"point_in_time_time_in_millis":0,"point_in_time_current":0,"suggest_total":0,"suggest_time_in_millis":0,"suggest_current":0,"request":{"dfs_pre_query":{"time_in_millis":0,"current":0,"total":0},"query":{"time_in_millis":14192577,"current":933641,"total":1569785},"fetch":{"time_in_millis":910479,"current":0,"total":1569785},"dfs_query":{"time_in_millis":0,"current":0,"total":0},"expand":{"time_in_millis":6039090,"current":0,"total":1569785},"can_match":{"time_in_millis":6768271,"current":0,"total":933641}}}}}}}

By disabling Operate completely via the helmchart, we can bring the scroll_current back to zero (within a minute). So it is clear that Optimize is the cause here.

In the pod logs of Optimize, we see a continuous stream of errors related to opensearch, trying to query, failing (because limit exeeding, so all shards fail), retrying (way too quick).

How can this be solved? It looks like we got in some situation where queries are executed in parallel; and without a back-off, it stays forever exceeding the limits.

2024-06-11 10:43:59.021  WARN 7 --- [   postimport_1] i.c.o.s.o.c.s.RichOpenSearchClient       : Error occurred when clearing the scroll with id [FGluY2x1ZGVfY29udGV4dF91dWlkDnF1ZXJ5VGhlbkZldGNoDRZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qVAWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qVEWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qVIWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qVMWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qVQWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qVUWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qVYWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qVcWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qVkWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qVgWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qVoWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qVsWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qVwWR1dET0drTmNUNW1iRHFyamF0cERZdw==]
2024-06-11 10:43:59.021 ERROR 7 --- [   postimport_1] i.c.o.s.o.c.s.RichOpenSearchClient       : Failed to search index: [operate-list-view-8.3.0_alias]! Reason: Shards failed executing request (request=org.opensearch.client.opensearch.core.SearchRequest@540864b0, failed shards=[org.opensearch.client.opensearch._types.ShardFailure@3f7acf64, org.opensearch.client.opensearch._types.ShardFailure@29566ca0, org.opensearch.client.opensearch._types.ShardFailure@9dedfd8, org.opensearch.client.opensearch._types.ShardFailure@1ee407b7, org.opensearch.client.opensearch._types.ShardFailure@770cfae0, org.opensearch.client.opensearch._types.ShardFailure@6d25abcd, org.opensearch.client.opensearch._types.ShardFailure@127a99ae, org.opensearch.client.opensearch._types.ShardFailure@2c1dfde9, org.opensearch.client.opensearch._types.ShardFailure@3cd6c970, org.opensearch.client.opensearch._types.ShardFailure@7ced66e4, org.opensearch.client.opensearch._types.ShardFailure@3a9c273e, org.opensearch.client.opensearch._types.ShardFailure@792d6731, org.opensearch.client.opensearch._types.ShardFailure@73babc2d, org.opensearch.client.opensearch._types.ShardFailure@64f72ef8, org.opensearch.client.opensearch._types.ShardFailure@3f89408f, org.opensearch.client.opensearch._types.ShardFailure@2d8fbca0])

io.camunda.operate.store.opensearch.client.OpenSearchFailedShardsException: Shards failed executing request (request=org.opensearch.client.opensearch.core.SearchRequest@540864b0, failed shards=[org.opensearch.client.opensearch._types.ShardFailure@3f7acf64, org.opensearch.client.opensearch._types.ShardFailure@29566ca0, org.opensearch.client.opensearch._types.ShardFailure@9dedfd8, org.opensearch.client.opensearch._types.ShardFailure@1ee407b7, org.opensearch.client.opensearch._types.ShardFailure@770cfae0, org.opensearch.client.opensearch._types.ShardFailure@6d25abcd, org.opensearch.client.opensearch._types.ShardFailure@127a99ae, org.opensearch.client.opensearch._types.ShardFailure@2c1dfde9, org.opensearch.client.opensearch._types.ShardFailure@3cd6c970, org.opensearch.client.opensearch._types.ShardFailure@7ced66e4, org.opensearch.client.opensearch._types.ShardFailure@3a9c273e, org.opensearch.client.opensearch._types.ShardFailure@792d6731, org.opensearch.client.opensearch._types.ShardFailure@73babc2d, org.opensearch.client.opensearch._types.ShardFailure@64f72ef8, org.opensearch.client.opensearch._types.ShardFailure@3f89408f, org.opensearch.client.opensearch._types.ShardFailure@2d8fbca0])
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.checkFailedShards(OpenSearchDocumentOperations.java:96) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.scrollWith(OpenSearchDocumentOperations.java:136) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.unsafeScrollWith(OpenSearchDocumentOperations.java:113) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.lambda$safeScrollWith$2(OpenSearchDocumentOperations.java:169) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.OpenSearchOperation.safe(OpenSearchOperation.java:48) [operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.safeScrollWith(OpenSearchDocumentOperations.java:167) [operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.safeScrollWith(OpenSearchDocumentOperations.java:159) [operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.scrollWith(OpenSearchDocumentOperations.java:203) [operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.zeebeimport.post.opensearch.OpensearchIncidentPostImportAction.queryData(OpensearchIncidentPostImportAction.java:560) [operate-importer-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.zeebeimport.post.opensearch.OpensearchIncidentPostImportAction.searchForInstances(OpensearchIncidentPostImportAction.java:195) [operate-importer-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.zeebeimport.post.AbstractIncidentPostImportAction.processPendingIncidents(AbstractIncidentPostImportAction.java:123) [operate-importer-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.zeebeimport.post.AbstractIncidentPostImportAction.performOneRound(AbstractIncidentPostImportAction.java:61) [operate-importer-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.zeebeimport.post.AbstractIncidentPostImportAction.run(AbstractIncidentPostImportAction.java:75) [operate-importer-8.5.0.jar!/:8.5.0]
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-6.1.5.jar!/:6.1.5]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]

2024-06-11 10:43:59.021 ERROR 7 --- [   postimport_1] c.o.z.p.AbstractIncidentPostImportAction : Exception occurred when performing post import for partition 1: Failed to search index: [operate-list-view-8.3.0_alias]! Reason: Shards failed executing request (request=org.opensearch.client.opensearch.core.SearchRequest@540864b0, failed shards=[org.opensearch.client.opensearch._types.ShardFailure@3f7acf64, org.opensearch.client.opensearch._types.ShardFailure@29566ca0, org.opensearch.client.opensearch._types.ShardFailure@9dedfd8, org.opensearch.client.opensearch._types.ShardFailure@1ee407b7, org.opensearch.client.opensearch._types.ShardFailure@770cfae0, org.opensearch.client.opensearch._types.ShardFailure@6d25abcd, org.opensearch.client.opensearch._types.ShardFailure@127a99ae, org.opensearch.client.opensearch._types.ShardFailure@2c1dfde9, org.opensearch.client.opensearch._types.ShardFailure@3cd6c970, org.opensearch.client.opensearch._types.ShardFailure@7ced66e4, org.opensearch.client.opensearch._types.ShardFailure@3a9c273e, org.opensearch.client.opensearch._types.ShardFailure@792d6731, org.opensearch.client.opensearch._types.ShardFailure@73babc2d, org.opensearch.client.opensearch._types.ShardFailure@64f72ef8, org.opensearch.client.opensearch._types.ShardFailure@3f89408f, org.opensearch.client.opensearch._types.ShardFailure@2d8fbca0]). Will be retried...

io.camunda.operate.exceptions.OperateRuntimeException: Failed to search index: [operate-list-view-8.3.0_alias]! Reason: Shards failed executing request (request=org.opensearch.client.opensearch.core.SearchRequest@540864b0, failed shards=[org.opensearch.client.opensearch._types.ShardFailure@3f7acf64, org.opensearch.client.opensearch._types.ShardFailure@29566ca0, org.opensearch.client.opensearch._types.ShardFailure@9dedfd8, org.opensearch.client.opensearch._types.ShardFailure@1ee407b7, org.opensearch.client.opensearch._types.ShardFailure@770cfae0, org.opensearch.client.opensearch._types.ShardFailure@6d25abcd, org.opensearch.client.opensearch._types.ShardFailure@127a99ae, org.opensearch.client.opensearch._types.ShardFailure@2c1dfde9, org.opensearch.client.opensearch._types.ShardFailure@3cd6c970, org.opensearch.client.opensearch._types.ShardFailure@7ced66e4, org.opensearch.client.opensearch._types.ShardFailure@3a9c273e, org.opensearch.client.opensearch._types.ShardFailure@792d6731, org.opensearch.client.opensearch._types.ShardFailure@73babc2d, org.opensearch.client.opensearch._types.ShardFailure@64f72ef8, org.opensearch.client.opensearch._types.ShardFailure@3f89408f, org.opensearch.client.opensearch._types.ShardFailure@2d8fbca0])
	at io.camunda.operate.store.opensearch.client.OpenSearchOperation.safe(OpenSearchOperation.java:54) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.safeScrollWith(OpenSearchDocumentOperations.java:167) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.safeScrollWith(OpenSearchDocumentOperations.java:159) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.scrollWith(OpenSearchDocumentOperations.java:203) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.zeebeimport.post.opensearch.OpensearchIncidentPostImportAction.queryData(OpensearchIncidentPostImportAction.java:560) ~[operate-importer-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.zeebeimport.post.opensearch.OpensearchIncidentPostImportAction.searchForInstances(OpensearchIncidentPostImportAction.java:195) ~[operate-importer-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.zeebeimport.post.AbstractIncidentPostImportAction.processPendingIncidents(AbstractIncidentPostImportAction.java:123) ~[operate-importer-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.zeebeimport.post.AbstractIncidentPostImportAction.performOneRound(AbstractIncidentPostImportAction.java:61) ~[operate-importer-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.zeebeimport.post.AbstractIncidentPostImportAction.run(AbstractIncidentPostImportAction.java:75) [operate-importer-8.5.0.jar!/:8.5.0]
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-6.1.5.jar!/:6.1.5]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: io.camunda.operate.store.opensearch.client.OpenSearchFailedShardsException: Shards failed executing request (request=org.opensearch.client.opensearch.core.SearchRequest@540864b0, failed shards=[org.opensearch.client.opensearch._types.ShardFailure@3f7acf64, org.opensearch.client.opensearch._types.ShardFailure@29566ca0, org.opensearch.client.opensearch._types.ShardFailure@9dedfd8, org.opensearch.client.opensearch._types.ShardFailure@1ee407b7, org.opensearch.client.opensearch._types.ShardFailure@770cfae0, org.opensearch.client.opensearch._types.ShardFailure@6d25abcd, org.opensearch.client.opensearch._types.ShardFailure@127a99ae, org.opensearch.client.opensearch._types.ShardFailure@2c1dfde9, org.opensearch.client.opensearch._types.ShardFailure@3cd6c970, org.opensearch.client.opensearch._types.ShardFailure@7ced66e4, org.opensearch.client.opensearch._types.ShardFailure@3a9c273e, org.opensearch.client.opensearch._types.ShardFailure@792d6731, org.opensearch.client.opensearch._types.ShardFailure@73babc2d, org.opensearch.client.opensearch._types.ShardFailure@64f72ef8, org.opensearch.client.opensearch._types.ShardFailure@3f89408f, org.opensearch.client.opensearch._types.ShardFailure@2d8fbca0])
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.checkFailedShards(OpenSearchDocumentOperations.java:96) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.scrollWith(OpenSearchDocumentOperations.java:136) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.unsafeScrollWith(OpenSearchDocumentOperations.java:113) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.lambda$safeScrollWith$2(OpenSearchDocumentOperations.java:169) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.OpenSearchOperation.safe(OpenSearchOperation.java:48) ~[operate-schema-8.5.0.jar!/:8.5.0]
	... 15 more

2024-06-11 10:44:01.141  WARN 7 --- [   postimport_1] i.c.o.s.o.c.s.RichOpenSearchClient       : Error occurred when clearing the scroll with id [FGluY2x1ZGVfY29udGV4dF91dWlkDnF1ZXJ5VGhlbkZldGNoDRZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qa4WR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qa8WR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qbAWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qbEWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qbIWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qbMWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qbQWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qbUWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qbYWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qbcWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qbgWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qbkWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qboWR1dET0drTmNUNW1iRHFyamF0cERZdw==]
2024-06-11 10:44:01.141 ERROR 7 --- [   postimport_1] i.c.o.s.o.c.s.RichOpenSearchClient       : Failed to search index: [operate-list-view-8.3.0_alias]! Reason: Shards failed executing request (request=org.opensearch.client.opensearch.core.SearchRequest@56af4990, failed shards=[org.opensearch.client.opensearch._types.ShardFailure@2357e6f2, org.opensearch.client.opensearch._types.ShardFailure@736d2484, org.opensearch.client.opensearch._types.ShardFailure@69b28977, org.opensearch.client.opensearch._types.ShardFailure@45fdbe18, org.opensearch.client.opensearch._types.ShardFailure@5981a023, org.opensearch.client.opensearch._types.ShardFailure@4352e693, org.opensearch.client.opensearch._types.ShardFailure@3ed636bf, org.opensearch.client.opensearch._types.ShardFailure@16735203, org.opensearch.client.opensearch._types.ShardFailure@42a76856, org.opensearch.client.opensearch._types.ShardFailure@169578ef, org.opensearch.client.opensearch._types.ShardFailure@19f9d74c, org.opensearch.client.opensearch._types.ShardFailure@15967d3a, org.opensearch.client.opensearch._types.ShardFailure@6c0a5206, org.opensearch.client.opensearch._types.ShardFailure@51e51d68, org.opensearch.client.opensearch._types.ShardFailure@671e3af, org.opensearch.client.opensearch._types.ShardFailure@202821f6])

io.camunda.operate.store.opensearch.client.OpenSearchFailedShardsException: Shards failed executing request (request=org.opensearch.client.opensearch.core.SearchRequest@56af4990, failed shards=[org.opensearch.client.opensearch._types.ShardFailure@2357e6f2, org.opensearch.client.opensearch._types.ShardFailure@736d2484, org.opensearch.client.opensearch._types.ShardFailure@69b28977, org.opensearch.client.opensearch._types.ShardFailure@45fdbe18, org.opensearch.client.opensearch._types.ShardFailure@5981a023, org.opensearch.client.opensearch._types.ShardFailure@4352e693, org.opensearch.client.opensearch._types.ShardFailure@3ed636bf, org.opensearch.client.opensearch._types.ShardFailure@16735203, org.opensearch.client.opensearch._types.ShardFailure@42a76856, org.opensearch.client.opensearch._types.ShardFailure@169578ef, org.opensearch.client.opensearch._types.ShardFailure@19f9d74c, org.opensearch.client.opensearch._types.ShardFailure@15967d3a, org.opensearch.client.opensearch._types.ShardFailure@6c0a5206, org.opensearch.client.opensearch._types.ShardFailure@51e51d68, org.opensearch.client.opensearch._types.ShardFailure@671e3af, org.opensearch.client.opensearch._types.ShardFailure@202821f6])
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.checkFailedShards(OpenSearchDocumentOperations.java:96) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.scrollWith(OpenSearchDocumentOperations.java:136) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.unsafeScrollWith(OpenSearchDocumentOperations.java:113) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.lambda$safeScrollWith$2(OpenSearchDocumentOperations.java:169) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.OpenSearchOperation.safe(OpenSearchOperation.java:48) [operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.safeScrollWith(OpenSearchDocumentOperations.java:167) [operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.safeScrollWith(OpenSearchDocumentOperations.java:159) [operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.scrollWith(OpenSearchDocumentOperations.java:203) [operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.zeebeimport.post.opensearch.OpensearchIncidentPostImportAction.queryData(OpensearchIncidentPostImportAction.java:560) [operate-importer-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.zeebeimport.post.opensearch.OpensearchIncidentPostImportAction.searchForInstances(OpensearchIncidentPostImportAction.java:189) [operate-importer-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.zeebeimport.post.AbstractIncidentPostImportAction.processPendingIncidents(AbstractIncidentPostImportAction.java:123) [operate-importer-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.zeebeimport.post.AbstractIncidentPostImportAction.performOneRound(AbstractIncidentPostImportAction.java:61) [operate-importer-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.zeebeimport.post.AbstractIncidentPostImportAction.run(AbstractIncidentPostImportAction.java:75) [operate-importer-8.5.0.jar!/:8.5.0]
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-6.1.5.jar!/:6.1.5]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]

2024-06-11 10:44:06.171  WARN 7 --- [   postimport_1] i.c.o.s.o.c.s.RichOpenSearchClient       : Error occurred when clearing the scroll with id [FGluY2x1ZGVfY29udGV4dF91dWlkDnF1ZXJ5VGhlbkZldGNoDRZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qmEWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qmIWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qmQWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qmUWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qmYWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qmcWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qmgWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qmkWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qmoWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qmsWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qmwWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qm0WR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qm4WR1dET0drTmNUNW1iRHFyamF0cERZdw==]
2024-06-11 10:44:06.171 ERROR 7 --- [   postimport_1] i.c.o.s.o.c.s.RichOpenSearchClient       : Failed to search index: [operate-list-view-8.3.0_alias]! Reason: Shards failed executing request (request=org.opensearch.client.opensearch.core.SearchRequest@1e0ccd97, failed shards=[org.opensearch.client.opensearch._types.ShardFailure@677b9d63, org.opensearch.client.opensearch._types.ShardFailure@6e1e66b8, org.opensearch.client.opensearch._types.ShardFailure@fd8dd1c, org.opensearch.client.opensearch._types.ShardFailure@77c0d905, org.opensearch.client.opensearch._types.ShardFailure@4cebd21b, org.opensearch.client.opensearch._types.ShardFailure@515d5698, org.opensearch.client.opensearch._types.ShardFailure@18244192, org.opensearch.client.opensearch._types.ShardFailure@69ddab62, org.opensearch.client.opensearch._types.ShardFailure@7bb4d748, org.opensearch.client.opensearch._types.ShardFailure@4066a730, org.opensearch.client.opensearch._types.ShardFailure@7af4567f, org.opensearch.client.opensearch._types.ShardFailure@7ef674f8, org.opensearch.client.opensearch._types.ShardFailure@2325e2f9, org.opensearch.client.opensearch._types.ShardFailure@560f8e8a, org.opensearch.client.opensearch._types.ShardFailure@e107d9a, org.opensearch.client.opensearch._types.ShardFailure@438c8914])


1 Like

I also experience a similar issue. Lots of errors in the operate logs with a few hundred instances. Am also using Opensearch. Lots of errors due to the import: “Error occurred when clearing the scroll with id”, and a rise in the amount of open scrolls.

I am also having this problem

Hi all, I reported this to the Operate team. Sounds like this is a bug, so they have opened an Issue on GitHub. You can keep up with it here:

Thank you for letting us know!

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.