Operate exeeds search.max_open_scroll_context on AWS using Opensearch

We are running camunda 10.0.4 via EKS, using an aws managed Opensearch 2.11.1 cluster, with 2 nodes, and have issues with Operate, that seems to saturate the scroll context of Opensearch.

All was running smooth, until somewhere last week a developer was running a test process, and did a bulk cancel via operate. Since then, the operate user interface is misbehaving. For a few days now, we see issues where Operate is overloading the Opensearch cluster (by exceeding the max_open_scroll_context=500 limit continuously). It sometimes shows process instance info, but more often not (then it helps refreshing 5 or 6 times until it suddenly shows something again). In operate, still there are about 100 instances shown to be still canceling (for 3 days now), that seem to be stuck now.

Increasing the limit of 500 open scroll contexts via AWS is not an option, since this limit is a hard enforced limit, managed by aws, not allowed to be changed in the cluster.

The exceeding of max_open_scroll_context can be seen in the /_nodes/stats/indices/search call, where both nodes have a “scroll_current”:500 ;

{"_nodes":{"total":2,"successful":2,"failed":0},"cluster_name":"opensearch","nodes":{"1Tdt2aBnSPuUA_n5XZQytg":{"timestamp":1718103102779,"name":"f4017cda3462a810ee8d3ee7b0845ab9","roles":["data","ingest","master","remote_cluster_client"],"indices":{"search":{"open_contexts":500,"query_total":30533548,"query_time_in_millis":5039314,"query_current":0,"fetch_total":1467197,"fetch_time_in_millis":676315,"fetch_current":0,"scroll_total":2520280,"scroll_time_in_millis":231961399070,"scroll_current":500,"point_in_time_total":0,"point_in_time_time_in_millis":0,"point_in_time_current":0,"suggest_total":0,"suggest_time_in_millis":0,"suggest_current":0,"request":{"dfs_pre_query":{"time_in_millis":0,"current":0,"total":0},"query":{"time_in_millis":18982857,"current":924843,"total":1555161},"fetch":{"time_in_millis":1055078,"current":0,"total":1555161},"dfs_query":{"time_in_millis":0,"current":0,"total":0},"expand":{"time_in_millis":5863815,"current":0,"total":1555161},"can_match":{"time_in_millis":6064734,"current":0,"total":924843}}}}},"GWDOGkNcT5mbDqrjatpDYw":{"timestamp":1718103102780,"name":"c69b1b5babd14f1aee46f5ca1f43faf6","roles":["data","ingest","master","remote_cluster_client"],"indices":{"search":{"open_contexts":500,"query_total":33557847,"query_time_in_millis":5325534,"query_current":0,"fetch_total":2183208,"fetch_time_in_millis":5861040,"fetch_current":0,"scroll_total":2556575,"scroll_time_in_millis":234506794018,"scroll_current":500,"point_in_time_total":0,"point_in_time_time_in_millis":0,"point_in_time_current":0,"suggest_total":0,"suggest_time_in_millis":0,"suggest_current":0,"request":{"dfs_pre_query":{"time_in_millis":0,"current":0,"total":0},"query":{"time_in_millis":14192577,"current":933641,"total":1569785},"fetch":{"time_in_millis":910479,"current":0,"total":1569785},"dfs_query":{"time_in_millis":0,"current":0,"total":0},"expand":{"time_in_millis":6039090,"current":0,"total":1569785},"can_match":{"time_in_millis":6768271,"current":0,"total":933641}}}}}}}

By disabling Operate completely via the helmchart, we can bring the scroll_current back to zero (within a minute). So it is clear that Optimize is the cause here.

In the pod logs of Optimize, we see a continuous stream of errors related to opensearch, trying to query, failing (because limit exeeding, so all shards fail), retrying (way too quick).

How can this be solved? It looks like we got in some situation where queries are executed in parallel; and without a back-off, it stays forever exceeding the limits.

2024-06-11 10:43:59.021  WARN 7 --- [   postimport_1] i.c.o.s.o.c.s.RichOpenSearchClient       : Error occurred when clearing the scroll with id [FGluY2x1ZGVfY29udGV4dF91dWlkDnF1ZXJ5VGhlbkZldGNoDRZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qVAWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qVEWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qVIWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qVMWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qVQWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qVUWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qVYWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qVcWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qVkWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qVgWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qVoWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qVsWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qVwWR1dET0drTmNUNW1iRHFyamF0cERZdw==]
2024-06-11 10:43:59.021 ERROR 7 --- [   postimport_1] i.c.o.s.o.c.s.RichOpenSearchClient       : Failed to search index: [operate-list-view-8.3.0_alias]! Reason: Shards failed executing request (request=org.opensearch.client.opensearch.core.SearchRequest@540864b0, failed shards=[org.opensearch.client.opensearch._types.ShardFailure@3f7acf64, org.opensearch.client.opensearch._types.ShardFailure@29566ca0, org.opensearch.client.opensearch._types.ShardFailure@9dedfd8, org.opensearch.client.opensearch._types.ShardFailure@1ee407b7, org.opensearch.client.opensearch._types.ShardFailure@770cfae0, org.opensearch.client.opensearch._types.ShardFailure@6d25abcd, org.opensearch.client.opensearch._types.ShardFailure@127a99ae, org.opensearch.client.opensearch._types.ShardFailure@2c1dfde9, org.opensearch.client.opensearch._types.ShardFailure@3cd6c970, org.opensearch.client.opensearch._types.ShardFailure@7ced66e4, org.opensearch.client.opensearch._types.ShardFailure@3a9c273e, org.opensearch.client.opensearch._types.ShardFailure@792d6731, org.opensearch.client.opensearch._types.ShardFailure@73babc2d, org.opensearch.client.opensearch._types.ShardFailure@64f72ef8, org.opensearch.client.opensearch._types.ShardFailure@3f89408f, org.opensearch.client.opensearch._types.ShardFailure@2d8fbca0])

io.camunda.operate.store.opensearch.client.OpenSearchFailedShardsException: Shards failed executing request (request=org.opensearch.client.opensearch.core.SearchRequest@540864b0, failed shards=[org.opensearch.client.opensearch._types.ShardFailure@3f7acf64, org.opensearch.client.opensearch._types.ShardFailure@29566ca0, org.opensearch.client.opensearch._types.ShardFailure@9dedfd8, org.opensearch.client.opensearch._types.ShardFailure@1ee407b7, org.opensearch.client.opensearch._types.ShardFailure@770cfae0, org.opensearch.client.opensearch._types.ShardFailure@6d25abcd, org.opensearch.client.opensearch._types.ShardFailure@127a99ae, org.opensearch.client.opensearch._types.ShardFailure@2c1dfde9, org.opensearch.client.opensearch._types.ShardFailure@3cd6c970, org.opensearch.client.opensearch._types.ShardFailure@7ced66e4, org.opensearch.client.opensearch._types.ShardFailure@3a9c273e, org.opensearch.client.opensearch._types.ShardFailure@792d6731, org.opensearch.client.opensearch._types.ShardFailure@73babc2d, org.opensearch.client.opensearch._types.ShardFailure@64f72ef8, org.opensearch.client.opensearch._types.ShardFailure@3f89408f, org.opensearch.client.opensearch._types.ShardFailure@2d8fbca0])
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.checkFailedShards(OpenSearchDocumentOperations.java:96) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.scrollWith(OpenSearchDocumentOperations.java:136) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.unsafeScrollWith(OpenSearchDocumentOperations.java:113) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.lambda$safeScrollWith$2(OpenSearchDocumentOperations.java:169) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.OpenSearchOperation.safe(OpenSearchOperation.java:48) [operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.safeScrollWith(OpenSearchDocumentOperations.java:167) [operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.safeScrollWith(OpenSearchDocumentOperations.java:159) [operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.scrollWith(OpenSearchDocumentOperations.java:203) [operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.zeebeimport.post.opensearch.OpensearchIncidentPostImportAction.queryData(OpensearchIncidentPostImportAction.java:560) [operate-importer-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.zeebeimport.post.opensearch.OpensearchIncidentPostImportAction.searchForInstances(OpensearchIncidentPostImportAction.java:195) [operate-importer-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.zeebeimport.post.AbstractIncidentPostImportAction.processPendingIncidents(AbstractIncidentPostImportAction.java:123) [operate-importer-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.zeebeimport.post.AbstractIncidentPostImportAction.performOneRound(AbstractIncidentPostImportAction.java:61) [operate-importer-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.zeebeimport.post.AbstractIncidentPostImportAction.run(AbstractIncidentPostImportAction.java:75) [operate-importer-8.5.0.jar!/:8.5.0]
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-6.1.5.jar!/:6.1.5]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]

2024-06-11 10:43:59.021 ERROR 7 --- [   postimport_1] c.o.z.p.AbstractIncidentPostImportAction : Exception occurred when performing post import for partition 1: Failed to search index: [operate-list-view-8.3.0_alias]! Reason: Shards failed executing request (request=org.opensearch.client.opensearch.core.SearchRequest@540864b0, failed shards=[org.opensearch.client.opensearch._types.ShardFailure@3f7acf64, org.opensearch.client.opensearch._types.ShardFailure@29566ca0, org.opensearch.client.opensearch._types.ShardFailure@9dedfd8, org.opensearch.client.opensearch._types.ShardFailure@1ee407b7, org.opensearch.client.opensearch._types.ShardFailure@770cfae0, org.opensearch.client.opensearch._types.ShardFailure@6d25abcd, org.opensearch.client.opensearch._types.ShardFailure@127a99ae, org.opensearch.client.opensearch._types.ShardFailure@2c1dfde9, org.opensearch.client.opensearch._types.ShardFailure@3cd6c970, org.opensearch.client.opensearch._types.ShardFailure@7ced66e4, org.opensearch.client.opensearch._types.ShardFailure@3a9c273e, org.opensearch.client.opensearch._types.ShardFailure@792d6731, org.opensearch.client.opensearch._types.ShardFailure@73babc2d, org.opensearch.client.opensearch._types.ShardFailure@64f72ef8, org.opensearch.client.opensearch._types.ShardFailure@3f89408f, org.opensearch.client.opensearch._types.ShardFailure@2d8fbca0]). Will be retried...

io.camunda.operate.exceptions.OperateRuntimeException: Failed to search index: [operate-list-view-8.3.0_alias]! Reason: Shards failed executing request (request=org.opensearch.client.opensearch.core.SearchRequest@540864b0, failed shards=[org.opensearch.client.opensearch._types.ShardFailure@3f7acf64, org.opensearch.client.opensearch._types.ShardFailure@29566ca0, org.opensearch.client.opensearch._types.ShardFailure@9dedfd8, org.opensearch.client.opensearch._types.ShardFailure@1ee407b7, org.opensearch.client.opensearch._types.ShardFailure@770cfae0, org.opensearch.client.opensearch._types.ShardFailure@6d25abcd, org.opensearch.client.opensearch._types.ShardFailure@127a99ae, org.opensearch.client.opensearch._types.ShardFailure@2c1dfde9, org.opensearch.client.opensearch._types.ShardFailure@3cd6c970, org.opensearch.client.opensearch._types.ShardFailure@7ced66e4, org.opensearch.client.opensearch._types.ShardFailure@3a9c273e, org.opensearch.client.opensearch._types.ShardFailure@792d6731, org.opensearch.client.opensearch._types.ShardFailure@73babc2d, org.opensearch.client.opensearch._types.ShardFailure@64f72ef8, org.opensearch.client.opensearch._types.ShardFailure@3f89408f, org.opensearch.client.opensearch._types.ShardFailure@2d8fbca0])
	at io.camunda.operate.store.opensearch.client.OpenSearchOperation.safe(OpenSearchOperation.java:54) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.safeScrollWith(OpenSearchDocumentOperations.java:167) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.safeScrollWith(OpenSearchDocumentOperations.java:159) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.scrollWith(OpenSearchDocumentOperations.java:203) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.zeebeimport.post.opensearch.OpensearchIncidentPostImportAction.queryData(OpensearchIncidentPostImportAction.java:560) ~[operate-importer-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.zeebeimport.post.opensearch.OpensearchIncidentPostImportAction.searchForInstances(OpensearchIncidentPostImportAction.java:195) ~[operate-importer-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.zeebeimport.post.AbstractIncidentPostImportAction.processPendingIncidents(AbstractIncidentPostImportAction.java:123) ~[operate-importer-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.zeebeimport.post.AbstractIncidentPostImportAction.performOneRound(AbstractIncidentPostImportAction.java:61) ~[operate-importer-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.zeebeimport.post.AbstractIncidentPostImportAction.run(AbstractIncidentPostImportAction.java:75) [operate-importer-8.5.0.jar!/:8.5.0]
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-6.1.5.jar!/:6.1.5]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: io.camunda.operate.store.opensearch.client.OpenSearchFailedShardsException: Shards failed executing request (request=org.opensearch.client.opensearch.core.SearchRequest@540864b0, failed shards=[org.opensearch.client.opensearch._types.ShardFailure@3f7acf64, org.opensearch.client.opensearch._types.ShardFailure@29566ca0, org.opensearch.client.opensearch._types.ShardFailure@9dedfd8, org.opensearch.client.opensearch._types.ShardFailure@1ee407b7, org.opensearch.client.opensearch._types.ShardFailure@770cfae0, org.opensearch.client.opensearch._types.ShardFailure@6d25abcd, org.opensearch.client.opensearch._types.ShardFailure@127a99ae, org.opensearch.client.opensearch._types.ShardFailure@2c1dfde9, org.opensearch.client.opensearch._types.ShardFailure@3cd6c970, org.opensearch.client.opensearch._types.ShardFailure@7ced66e4, org.opensearch.client.opensearch._types.ShardFailure@3a9c273e, org.opensearch.client.opensearch._types.ShardFailure@792d6731, org.opensearch.client.opensearch._types.ShardFailure@73babc2d, org.opensearch.client.opensearch._types.ShardFailure@64f72ef8, org.opensearch.client.opensearch._types.ShardFailure@3f89408f, org.opensearch.client.opensearch._types.ShardFailure@2d8fbca0])
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.checkFailedShards(OpenSearchDocumentOperations.java:96) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.scrollWith(OpenSearchDocumentOperations.java:136) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.unsafeScrollWith(OpenSearchDocumentOperations.java:113) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.lambda$safeScrollWith$2(OpenSearchDocumentOperations.java:169) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.OpenSearchOperation.safe(OpenSearchOperation.java:48) ~[operate-schema-8.5.0.jar!/:8.5.0]
	... 15 more

2024-06-11 10:44:01.141  WARN 7 --- [   postimport_1] i.c.o.s.o.c.s.RichOpenSearchClient       : Error occurred when clearing the scroll with id [FGluY2x1ZGVfY29udGV4dF91dWlkDnF1ZXJ5VGhlbkZldGNoDRZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qa4WR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qa8WR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qbAWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qbEWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qbIWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qbMWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qbQWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qbUWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qbYWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qbcWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qbgWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qbkWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qboWR1dET0drTmNUNW1iRHFyamF0cERZdw==]
2024-06-11 10:44:01.141 ERROR 7 --- [   postimport_1] i.c.o.s.o.c.s.RichOpenSearchClient       : Failed to search index: [operate-list-view-8.3.0_alias]! Reason: Shards failed executing request (request=org.opensearch.client.opensearch.core.SearchRequest@56af4990, failed shards=[org.opensearch.client.opensearch._types.ShardFailure@2357e6f2, org.opensearch.client.opensearch._types.ShardFailure@736d2484, org.opensearch.client.opensearch._types.ShardFailure@69b28977, org.opensearch.client.opensearch._types.ShardFailure@45fdbe18, org.opensearch.client.opensearch._types.ShardFailure@5981a023, org.opensearch.client.opensearch._types.ShardFailure@4352e693, org.opensearch.client.opensearch._types.ShardFailure@3ed636bf, org.opensearch.client.opensearch._types.ShardFailure@16735203, org.opensearch.client.opensearch._types.ShardFailure@42a76856, org.opensearch.client.opensearch._types.ShardFailure@169578ef, org.opensearch.client.opensearch._types.ShardFailure@19f9d74c, org.opensearch.client.opensearch._types.ShardFailure@15967d3a, org.opensearch.client.opensearch._types.ShardFailure@6c0a5206, org.opensearch.client.opensearch._types.ShardFailure@51e51d68, org.opensearch.client.opensearch._types.ShardFailure@671e3af, org.opensearch.client.opensearch._types.ShardFailure@202821f6])

io.camunda.operate.store.opensearch.client.OpenSearchFailedShardsException: Shards failed executing request (request=org.opensearch.client.opensearch.core.SearchRequest@56af4990, failed shards=[org.opensearch.client.opensearch._types.ShardFailure@2357e6f2, org.opensearch.client.opensearch._types.ShardFailure@736d2484, org.opensearch.client.opensearch._types.ShardFailure@69b28977, org.opensearch.client.opensearch._types.ShardFailure@45fdbe18, org.opensearch.client.opensearch._types.ShardFailure@5981a023, org.opensearch.client.opensearch._types.ShardFailure@4352e693, org.opensearch.client.opensearch._types.ShardFailure@3ed636bf, org.opensearch.client.opensearch._types.ShardFailure@16735203, org.opensearch.client.opensearch._types.ShardFailure@42a76856, org.opensearch.client.opensearch._types.ShardFailure@169578ef, org.opensearch.client.opensearch._types.ShardFailure@19f9d74c, org.opensearch.client.opensearch._types.ShardFailure@15967d3a, org.opensearch.client.opensearch._types.ShardFailure@6c0a5206, org.opensearch.client.opensearch._types.ShardFailure@51e51d68, org.opensearch.client.opensearch._types.ShardFailure@671e3af, org.opensearch.client.opensearch._types.ShardFailure@202821f6])
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.checkFailedShards(OpenSearchDocumentOperations.java:96) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.scrollWith(OpenSearchDocumentOperations.java:136) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.unsafeScrollWith(OpenSearchDocumentOperations.java:113) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.lambda$safeScrollWith$2(OpenSearchDocumentOperations.java:169) ~[operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.OpenSearchOperation.safe(OpenSearchOperation.java:48) [operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.safeScrollWith(OpenSearchDocumentOperations.java:167) [operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.safeScrollWith(OpenSearchDocumentOperations.java:159) [operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.store.opensearch.client.sync.OpenSearchDocumentOperations.scrollWith(OpenSearchDocumentOperations.java:203) [operate-schema-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.zeebeimport.post.opensearch.OpensearchIncidentPostImportAction.queryData(OpensearchIncidentPostImportAction.java:560) [operate-importer-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.zeebeimport.post.opensearch.OpensearchIncidentPostImportAction.searchForInstances(OpensearchIncidentPostImportAction.java:189) [operate-importer-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.zeebeimport.post.AbstractIncidentPostImportAction.processPendingIncidents(AbstractIncidentPostImportAction.java:123) [operate-importer-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.zeebeimport.post.AbstractIncidentPostImportAction.performOneRound(AbstractIncidentPostImportAction.java:61) [operate-importer-8.5.0.jar!/:8.5.0]
	at io.camunda.operate.zeebeimport.post.AbstractIncidentPostImportAction.run(AbstractIncidentPostImportAction.java:75) [operate-importer-8.5.0.jar!/:8.5.0]
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-6.1.5.jar!/:6.1.5]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]

2024-06-11 10:44:06.171  WARN 7 --- [   postimport_1] i.c.o.s.o.c.s.RichOpenSearchClient       : Error occurred when clearing the scroll with id [FGluY2x1ZGVfY29udGV4dF91dWlkDnF1ZXJ5VGhlbkZldGNoDRZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qmEWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qmIWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qmQWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qmUWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qmYWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qmcWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qmgWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qmkWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qmoWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qmsWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qmwWR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qm0WR1dET0drTmNUNW1iRHFyamF0cERZdxZEaXphaVJnQlFNeThSWTBjNDFKM1NRAAAAAAF2qm4WR1dET0drTmNUNW1iRHFyamF0cERZdw==]
2024-06-11 10:44:06.171 ERROR 7 --- [   postimport_1] i.c.o.s.o.c.s.RichOpenSearchClient       : Failed to search index: [operate-list-view-8.3.0_alias]! Reason: Shards failed executing request (request=org.opensearch.client.opensearch.core.SearchRequest@1e0ccd97, failed shards=[org.opensearch.client.opensearch._types.ShardFailure@677b9d63, org.opensearch.client.opensearch._types.ShardFailure@6e1e66b8, org.opensearch.client.opensearch._types.ShardFailure@fd8dd1c, org.opensearch.client.opensearch._types.ShardFailure@77c0d905, org.opensearch.client.opensearch._types.ShardFailure@4cebd21b, org.opensearch.client.opensearch._types.ShardFailure@515d5698, org.opensearch.client.opensearch._types.ShardFailure@18244192, org.opensearch.client.opensearch._types.ShardFailure@69ddab62, org.opensearch.client.opensearch._types.ShardFailure@7bb4d748, org.opensearch.client.opensearch._types.ShardFailure@4066a730, org.opensearch.client.opensearch._types.ShardFailure@7af4567f, org.opensearch.client.opensearch._types.ShardFailure@7ef674f8, org.opensearch.client.opensearch._types.ShardFailure@2325e2f9, org.opensearch.client.opensearch._types.ShardFailure@560f8e8a, org.opensearch.client.opensearch._types.ShardFailure@e107d9a, org.opensearch.client.opensearch._types.ShardFailure@438c8914])


I also experience a similar issue. Lots of errors in the operate logs with a few hundred instances. Am also using Opensearch. Lots of errors due to the import: “Error occurred when clearing the scroll with id”, and a rise in the amount of open scrolls.