Zeebe Operate 1.1.2 Performance Issues

ssach · June 28, 2023, 7:45am

We are using Zeebe Operate ver 1.1.2 and are seeing severe performance issues. We have a single node elastic search cluster dedicated for Zeebe (ver 1.0.0) and Operate.
We have more than 1.2 lakh instances being displayed on Operate.

Sometimes it takes a lot of time to

show all the deployed WFs
open and load a workflow instance
login to Zeebe Operate.

Following are some of the exceptions we have seen in Operate Logs:

search_context_missing_exception in Operate Logs

2023-06-22 15:03:03.659 ERROR 7 — [-8080-exec-1905] o.a.c.c.C.[.[.[.[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is ElasticsearchStatusException[Elasticsearch exception [type=search_phase_execution_exception, reason=all shards failed]]; nested: ElasticsearchException[Elasticsearch exception [type=search_context_missing_exception, reason=No search context found for id [574710291]]];]

Operate Import Exception in Operate Logs

2023-06-23 00:23:35.013 ERROR 7 — [ import_3] i.c.o.z.ImportJob : io.camunda.operate.exceptions.PersistenceException: Error when processing bulk request against Elasticsearch: 30,000 milliseconds timeout on connection http-outgoing-29 [ACTIVE]

io.camunda.operate.exceptions.PersistenceException: io.camunda.operate.exceptions.PersistenceException: Error when processing bulk request against Elasticsearch: 30,000 milliseconds timeout on connection http-outgoing-29 [ACTIVE]
at io.camunda.operate.zeebeimport.AbstractImportBatchProcessor.performImport(AbstractImportBatchProcessor.java:34) ~[operate-importer-common-1.1.2.jar!/:?]
at io.camunda.operate.zeebeimport.ImportJob.processOneIndexBatch(ImportJob.java:116) ~[operate-importer-1.1.2.jar!/:?]
at io.camunda.operate.zeebeimport.ImportJob.call(ImportJob.java:80) ~[operate-importer-1.1.2.jar!/:?]
at io.camunda.operate.zeebeimport.RecordsReader.lambda$scheduleImport$1(RecordsReader.java:217) ~[operate-importer-1.1.2.jar!/:?]
at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: io.camunda.operate.exceptions.PersistenceException: Error when processing bulk request against Elasticsearch: 30,000 milliseconds timeout on connection http-outgoing-29 [ACTIVE]
at io.camunda.operate.util.ElasticsearchUtil.processBulkRequest(ElasticsearchUtil.java:254) ~[operate-els-schema-1.1.2.jar!/:?]
at io.camunda.operate.util.ElasticsearchUtil.processBulkRequest(ElasticsearchUtil.java:233) ~[operate-els-schema-1.1.2.jar!/:?]

Delete Scroll Exception in Operate logs

2023-06-23 08:12:21.089 ERROR 7 — [-8080-exec-2742] o.a.c.c.C.[.[.[.[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is ElasticsearchStatusException[Unable to parse response body]; nested: ResponseException[method [DELETE], host [http://elasticsearch:9200], URI [/_search/scroll], status line [HTTP/1.1 404 Not Found]
{“succeeded”:true,“num_freed”:0}];] with root cause

org.elasticsearch.client.ResponseException: method [DELETE], host [http://elasticsearch:9200], URI [/_search/scroll], status line [HTTP/1.1 404 Not Found]
{“succeeded”:true,“num_freed”:0}
at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:326) ~[elasticsearch-rest-client-7.13.2.jar!/:7.13.2]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296) ~[elasticsearch-rest-client-7.13.2.jar!/:7.13.2]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270) ~[elasticsearch-rest-client-7.13.2.jar!/:7.13.2]
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1654) ~[elasticsearch-rest-high-level-client-7.13.2.jar!/:7.13.2]

ElasticsearchException Exception in Operate logs

org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=es_rejected_execution_exception, reason=rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@4cb9ae40 on QueueResizingEsThreadPoolExecutor[name = elasticsearch-0/search, queue capacity = 1000, min queue capacity = 1000, max queue capacity = 1000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 7.8ms, adjustment amount = 50, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@57f4d46b[Running, pool size = 10, active threads = 10, queued tasks = 1000, completed tasks = 653786320]]]
at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:485) ~[elasticsearch-7.13.2.jar!/:7.13.2]
at org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:396) ~[elasticsearch-7.13.2.jar!/:7.13.2]

Please let us know how to fix these issues and improve Zeebe Operate 1.1.2 Performance.

Would upgrading to latest version of Zeebe Operate fix these Performance issues?

Any suggestions on these lines would be really helpful.

jonathan.lukas · June 30, 2023, 4:45am

Hello @ssach ,

all of these logs seem to be related to elasticsearch. Do you perform a regular cleanup on the data?

Are you planning to upgrade to 8.2.x anytime soon?

Jonathan

ssach · June 30, 2023, 6:43am

Thanks for replying Jonathan. But not all are Elastic Search exceptions as few are Operate Import exceptions as you can see above. Unfortunately, we have a data retention policy of 8 months and cannot perform any clean up until August 2023. We are working on upgrading to 8.0.0 in parallel; but until we complete the upgrade, we want currently running Zeebe Operate ver 1.1.2 to run fine without any performance issues.

jonathan.lukas · June 30, 2023, 8:38am

Hello @ssach ,

what version of Elasticsearch are you using? Is the Elasticsearch instance shared with other applications? To me, it looks like Elasticsearch is slow and sometimes not responsive:

search_context_missing_exception in Operate Logs

There was no search context found is probably related to Elasticsearch performance.

Operate Import Exception in Operate Logs

There was a request timeout after 30 seconds, I would assume that either the network was slow or Elasticsearch was slow

Delete Scroll Exception in Operate logs

It seems that the scroll api was called with a DELETE method which is not possible. Does the stacktrace mention the origin for the request in the Operate code?

ElasticsearchException Exception in Operate logs

Here, it seems that the indexing execution cannot keep up with the requests:

ssach · June 30, 2023, 9:45am

We are using Elastic Search version 7.13.2 and as mentioned earlier it’s a single node cluster dedicated for Zeebe (ver 1.0.0) and Operate.
I checked Elasticsearch logs as well and didn’t see any errors; so I assume Elasticsearch is running fine.
We don’t have any control on the operate 1.1.2 code and not very sure how Scroll API is getting used inside operate 1.1.2 code; as we are not maintaining it. However, have attached the stack trace for the Delete Scroll Exception in Operate logs for your reference below.

Is there any recommended configurations for Zeebe Operate 1.1.2 alongwith Elastic Search version 7.13.2 for better performance?
Does Zeebe Operate 1.1.2 have any performance bottlenecks?

ssach · June 30, 2023, 11:00am

DeleteScrollException complete stack trace for reference.

2023-06-28 11:51:16.861 ERROR 7 — [-8080-exec-4824] o.a.c.c.C.[.[.[.[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is ElasticsearchStatusException[Unable to parse response body]; nested: ResponseException[method [DELETE], host [http://elasticsearch:9200], URI [/_search/scroll], status line [HTTP/1.1 404 Not Found]
{“succeeded”:true,“num_freed”:0}];] with root cause

org.elasticsearch.client.ResponseException: method [DELETE], host [http://elasticsearch:9200], URI [/_search/scroll], status line [HTTP/1.1 404 Not Found]
{“succeeded”:true,“num_freed”:0}
at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:326) ~[elasticsearch-rest-client-7.13.2.jar!/:7.13.2]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296) ~[elasticsearch-rest-client-7.13.2.jar!/:7.13.2]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270) ~[elasticsearch-rest-client-7.13.2.jar!/:7.13.2]
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1654) ~[elasticsearch-rest-high-level-client-7.13.2.jar!/:7.13.2]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1624) ~[elasticsearch-rest-high-level-client-7.13.2.jar!/:7.13.2]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1594) ~[elasticsearch-rest-high-level-client-7.13.2.jar!/:7.13.2]
at org.elasticsearch.client.RestHighLevelClient.clearScroll(RestHighLevelClient.java:1266) ~[elasticsearch-rest-high-level-client-7.13.2.jar!/:7.13.2]
at io.camunda.operate.util.ElasticsearchUtil.clearScroll(ElasticsearchUtil.java:411) ~[operate-els-schema-1.1.2.jar!/:?]
at io.camunda.operate.util.ElasticsearchUtil.scroll(ElasticsearchUtil.java:363) ~[operate-els-schema-1.1.2.jar!/:?]
at io.camunda.operate.util.ElasticsearchUtil.scroll(ElasticsearchUtil.java:322) ~[operate-els-schema-1.1.2.jar!/:?]
at io.camunda.operate.util.ElasticsearchUtil.scroll(ElasticsearchUtil.java:317) ~[operate-els-schema-1.1.2.jar!/:?]
at io.camunda.operate.webapp.es.reader.AbstractReader.scroll(AbstractReader.java:28) ~[classes!/:?]
at io.camunda.operate.webapp.es.reader.SequenceFlowReader.getSequenceFlowsByProcessInstanceKey(SequenceFlowReader.java:44) ~[classes!/:?]
at io.camunda.operate.webapp.rest.ProcessInstanceRestService.querySequenceFlowsByProcessInstanceId(ProcessInstanceRestService.java:184) ~[classes!/:?]
at jdk.internal.reflect.GeneratedMethodAccessor366.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197) ~[spring-web-5.3.8.jar!/:5.3.8]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141) ~[spring-web-5.3.8.jar!/:5.3.8]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106) ~[spring-webmvc-5.3.8.jar!/:5.3.8]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:894) ~[spring-webmvc-5.3.8.jar!/:5.3.8]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) ~[spring-webmvc-5.3.8.jar!/:5.3.8]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.3.8.jar!/:5.3.8]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1063) ~[spring-webmvc-5.3.8.jar!/:5.3.8]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) ~[spring-webmvc-5.3.8.jar!/:5.3.8]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.3.8.jar!/:5.3.8]
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) ~[spring-webmvc-5.3.8.jar!/:5.3.8]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:655) ~[tomcat-embed-core-9.0.48.jar!/:4.0.1]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.3.8.jar!/:5.3.8]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) ~[tomcat-embed-core-9.0.48.jar!/:4.0.1]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:228) ~[tomcat-embed-core-9.0.48.jar!/:?]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163) ~[tomcat-embed-core-9.0.48.jar!/:?]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.48.jar!/:?]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190) ~[tomcat-embed-core-9.0.48.jar!/:?]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163) ~[tomcat-embed-core-9.0.48.jar!/:?]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:327) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:121) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:126) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:81) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:105) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:218) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:212) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:103) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:89) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at io.camunda.operate.webapp.security.CSRFProtectable$1.doFilterInternal(CSRFProtectable.java:51) ~[classes!/:?]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.8.jar!/:5.3.8]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:117) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.8.jar!/:5.3.8]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.8.jar!/:5.3.8]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.8.jar!/:5.3.8]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:211) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183) ~[spring-security-web-5.5.1.jar!/:5.5.1]
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) ~[spring-web-5.3.8.jar!/:5.3.8]
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) ~[spring-web-5.3.8.jar!/:5.3.8]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190) ~[tomcat-embed-core-9.0.48.jar!/:?]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163) ~[tomcat-embed-core-9.0.48.jar!/:?]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.3.8.jar!/:5.3.8]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.8.jar!/:5.3.8]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190) ~[tomcat-embed-core-9.0.48.jar!/:?]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163) ~[tomcat-embed-core-9.0.48.jar!/:?]
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.3.8.jar!/:5.3.8]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.8.jar!/:5.3.8]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190) ~[tomcat-embed-core-9.0.48.jar!/:?]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163) ~[tomcat-embed-core-9.0.48.jar!/:?]
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) ~[spring-boot-actuator-2.5.2.jar!/:2.5.2]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.8.jar!/:5.3.8]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190) ~[tomcat-embed-core-9.0.48.jar!/:?]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163) ~[tomcat-embed-core-9.0.48.jar!/:?]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.3.8.jar!/:5.3.8]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.8.jar!/:5.3.8]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190) ~[tomcat-embed-core-9.0.48.jar!/:?]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163) ~[tomcat-embed-core-9.0.48.jar!/:?]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) [tomcat-embed-core-9.0.48.jar!/:?]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) [tomcat-embed-core-9.0.48.jar!/:?]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542) [tomcat-embed-core-9.0.48.jar!/:?]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) [tomcat-embed-core-9.0.48.jar!/:?]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.48.jar!/:?]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) [tomcat-embed-core-9.0.48.jar!/:?]
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:764) [tomcat-embed-core-9.0.48.jar!/:?]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357) [tomcat-embed-core-9.0.48.jar!/:?]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382) [tomcat-embed-core-9.0.48.jar!/:?]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.48.jar!/:?]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893) [tomcat-embed-core-9.0.48.jar!/:?]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723) [tomcat-embed-core-9.0.48.jar!/:?]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.48.jar!/:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.48.jar!/:?]
at java.lang.Thread.run(Unknown Source) [?:?]

jonathan.lukas · June 30, 2023, 11:26am

Hello @ssach ,

thank you for the details.

One last question on your setup: Why are you on this old patch version? For Operate, there are newer versions available:

Docker.

I also forwarded the thread internally. Also, if you run in production, you should be a customer and be able to use Camunda Support?

Jonathan

ssach · June 30, 2023, 11:40am

Actually we are running this in non-production setup and since we started out development with Zeebe Operate 1.1.2 we have continued to use the same until we upgrade to 8.0.0 or above.

Please do let me know once you hear back from your internal team on solutions/recommendations for these performance issues.

jonathan.lukas · June 30, 2023, 12:05pm

Hello @ssach ,

I got the feedback that the used version is deprecated since 2021. If you cannot migrate to 8.2.x right away, please look into patching Operate to a later version and see if the issue is resolved.

Jonathan

ssach · June 30, 2023, 12:18pm

Ok thank you. Which operate versions are still not deprecated?

jonathan.lukas · June 30, 2023, 12:25pm

Hello @ssach ,

all 1.x versions are out of support, 8.0 is supported until October 2023:

Jonathan