Errors in connectors when connecting to operate to fetch process definitions after helm upgrade

Initially everything starts up as expected. Then I do a helm upgrade and the camunda services restart (actually nothing changes but the helm chart version is different each time due to out CI/CD settings)

Then i start to receive errors like

2024-04-27T09:05:26.362Z ERROR 1 --- [   scheduling-1] i.c.c.r.i.i.ProcessDefinitionImporter    : Failed to import process definitions

io.camunda.identity.sdk.impl.rest.exception.RestException: request failed with status code '401' and body '{"error":"unauthorized_client","error_description":"Invalid client or Invalid client credentials"}'
	at io.camunda.identity.sdk.impl.rest.RestClient.send(RestClient.java:123)
	at io.camunda.identity.sdk.impl.rest.RestClient.request(RestClient.java:105)
	at io.camunda.identity.sdk.impl.keycloak.KeycloakAuthentication.requestFreshToken(KeycloakAuthentication.java:91)
	at io.camunda.identity.sdk.authentication.AbstractAuthentication.requestToken(AbstractAuthentication.java:76)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at io.camunda.identity.sdk.annotation.AnnotationProcessor.lambda$apply$0(AnnotationProcessor.java:33)
	at jdk.proxy2/jdk.proxy2.$Proxy100.requestToken(Unknown Source)
	at io.camunda.common.auth.SelfManagedAuthentication.getIdentityToken(SelfManagedAuthentication.java:31)
	at io.camunda.common.auth.SelfManagedAuthentication.generateToken(SelfManagedAuthentication.java:23)
	at io.camunda.common.auth.JwtAuthentication.getTokenHeader(JwtAuthentication.java:30)
	at io.camunda.common.http.DefaultHttpClient.retrieveToken(DefaultHttpClient.java:207)
	at io.camunda.common.http.DefaultHttpClient.post(DefaultHttpClient.java:148)
	at io.camunda.operate.CamundaOperateClient.searchProcessDefinitionResults(CamundaOperateClient.java:46)
	at io.camunda.connector.runtime.inbound.importer.ProcessDefinitionSearch.query(ProcessDefinitionSearch.java:72)
	at io.camunda.connector.runtime.inbound.importer.ProcessDefinitionImporter.scheduleImport(ProcessDefinitionImporter.java:55)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at org.springframework.scheduling.support.ScheduledMethodRunnable.runInternal(ScheduledMethodRunnable.java:130)
	at org.springframework.scheduling.support.ScheduledMethodRunnable.lambda$run$2(ScheduledMethodRunnable.java:124)
	at io.micrometer.observation.Observation.observe(Observation.java:499)
	at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:124)
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)

every 5 seconds, the connectors pods become unready.

I tried to restart the connectors pods (scaled to 0 and then back to 2), but they stay log the same error every 5 sec. and stay in not ready state.

But restarting the identity pod makes the connectors connect succesfully and work fine until the helm is upgraded again.
It still logs an error (different) every 5 minutes (instead of 5 sec), but works fine and actually fetces process definition updates from operate succesfully as soon as whey are made

2024-04-27T10:22:34.916Z ERROR 1 --- [   scheduling-1] i.camunda.common.http.DefaultHttpClient  : Failed POST with responseType class io.camunda.operate.model.SearchResult, parameterType class io.camunda.operate.model.ProcessDefinition, selector io.camunda.operate.model.SearchResult<io.camunda.operate.model.ProcessDefinition>, body io.camunda.operate.search.SearchQuery@414b067f due to Response not successful: 401
2024-04-27T10:22:34.916Z ERROR 1 --- [   scheduling-1] i.c.c.r.i.i.ProcessDefinitionImporter    : Failed to import process definitions

io.camunda.common.exception.SdkException: io.camunda.common.exception.SdkException: Response not successful: 401
	at io.camunda.common.http.DefaultHttpClient.post(DefaultHttpClient.java:162)
	at io.camunda.operate.CamundaOperateClient.searchProcessDefinitionResults(CamundaOperateClient.java:46)
	at io.camunda.connector.runtime.inbound.importer.ProcessDefinitionSearch.query(ProcessDefinitionSearch.java:72)
	at io.camunda.connector.runtime.inbound.importer.ProcessDefinitionImporter.scheduleImport(ProcessDefinitionImporter.java:55)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at org.springframework.scheduling.support.ScheduledMethodRunnable.runInternal(ScheduledMethodRunnable.java:130)
	at org.springframework.scheduling.support.ScheduledMethodRunnable.lambda$run$2(ScheduledMethodRunnable.java:124)
	at io.micrometer.observation.Observation.observe(Observation.java:499)
	at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:124)
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: io.camunda.common.exception.SdkException: Response not successful: 401
	at io.camunda.common.http.DefaultHttpClient.parseAndRetry(DefaultHttpClient.java:295)
	at io.camunda.common.http.DefaultHttpClient.post(DefaultHttpClient.java:153)
	... 16 common frames omitted

Here is the full log after the helm upgrade before I restarted identity
connectors.yaml (176.6 KB)

and here is the part of the log after identity was restarted and connectors started up successfully
connectors_end.yaml (123.7 KB)

Hi, were you able to solve this problem?

no, but it didn’t happened again

Well, on my side, it was resolved by updating the connectors to version 8.5.2

I described 2 problems - connector was unable to connect to operate after a helm upgrade till I manually restarted identity pod - it never happened again.

Second problem - connector was posting every 5 min into log about connection problems with operate, but worked fine (fetched new process definitions successfully when they were deployed) - this problem is gone after update to camunda 8.5.1 | connectors 8.5.2. But as far as I can see it was a bug of operate when it cound not generate post responses with emty body (no new processes)