Initially everything starts up as expected. Then I do a helm upgrade and the camunda services restart (actually nothing changes but the helm chart version is different each time due to out CI/CD settings)
Then i start to receive errors like
2024-04-27T09:05:26.362Z ERROR 1 --- [ scheduling-1] i.c.c.r.i.i.ProcessDefinitionImporter : Failed to import process definitions
io.camunda.identity.sdk.impl.rest.exception.RestException: request failed with status code '401' and body '{"error":"unauthorized_client","error_description":"Invalid client or Invalid client credentials"}'
at io.camunda.identity.sdk.impl.rest.RestClient.send(RestClient.java:123)
at io.camunda.identity.sdk.impl.rest.RestClient.request(RestClient.java:105)
at io.camunda.identity.sdk.impl.keycloak.KeycloakAuthentication.requestFreshToken(KeycloakAuthentication.java:91)
at io.camunda.identity.sdk.authentication.AbstractAuthentication.requestToken(AbstractAuthentication.java:76)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at io.camunda.identity.sdk.annotation.AnnotationProcessor.lambda$apply$0(AnnotationProcessor.java:33)
at jdk.proxy2/jdk.proxy2.$Proxy100.requestToken(Unknown Source)
at io.camunda.common.auth.SelfManagedAuthentication.getIdentityToken(SelfManagedAuthentication.java:31)
at io.camunda.common.auth.SelfManagedAuthentication.generateToken(SelfManagedAuthentication.java:23)
at io.camunda.common.auth.JwtAuthentication.getTokenHeader(JwtAuthentication.java:30)
at io.camunda.common.http.DefaultHttpClient.retrieveToken(DefaultHttpClient.java:207)
at io.camunda.common.http.DefaultHttpClient.post(DefaultHttpClient.java:148)
at io.camunda.operate.CamundaOperateClient.searchProcessDefinitionResults(CamundaOperateClient.java:46)
at io.camunda.connector.runtime.inbound.importer.ProcessDefinitionSearch.query(ProcessDefinitionSearch.java:72)
at io.camunda.connector.runtime.inbound.importer.ProcessDefinitionImporter.scheduleImport(ProcessDefinitionImporter.java:55)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.scheduling.support.ScheduledMethodRunnable.runInternal(ScheduledMethodRunnable.java:130)
at org.springframework.scheduling.support.ScheduledMethodRunnable.lambda$run$2(ScheduledMethodRunnable.java:124)
at io.micrometer.observation.Observation.observe(Observation.java:499)
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:124)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
every 5 seconds, the connectors pods become unready.
I tried to restart the connectors pods (scaled to 0 and then back to 2), but they stay log the same error every 5 sec. and stay in not ready state.
But restarting the identity pod makes the connectors connect succesfully and work fine until the helm is upgraded again.
It still logs an error (different) every 5 minutes (instead of 5 sec), but works fine and actually fetces process definition updates from operate succesfully as soon as whey are made
2024-04-27T10:22:34.916Z ERROR 1 --- [ scheduling-1] i.camunda.common.http.DefaultHttpClient : Failed POST with responseType class io.camunda.operate.model.SearchResult, parameterType class io.camunda.operate.model.ProcessDefinition, selector io.camunda.operate.model.SearchResult<io.camunda.operate.model.ProcessDefinition>, body io.camunda.operate.search.SearchQuery@414b067f due to Response not successful: 401
2024-04-27T10:22:34.916Z ERROR 1 --- [ scheduling-1] i.c.c.r.i.i.ProcessDefinitionImporter : Failed to import process definitions
io.camunda.common.exception.SdkException: io.camunda.common.exception.SdkException: Response not successful: 401
at io.camunda.common.http.DefaultHttpClient.post(DefaultHttpClient.java:162)
at io.camunda.operate.CamundaOperateClient.searchProcessDefinitionResults(CamundaOperateClient.java:46)
at io.camunda.connector.runtime.inbound.importer.ProcessDefinitionSearch.query(ProcessDefinitionSearch.java:72)
at io.camunda.connector.runtime.inbound.importer.ProcessDefinitionImporter.scheduleImport(ProcessDefinitionImporter.java:55)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.scheduling.support.ScheduledMethodRunnable.runInternal(ScheduledMethodRunnable.java:130)
at org.springframework.scheduling.support.ScheduledMethodRunnable.lambda$run$2(ScheduledMethodRunnable.java:124)
at io.micrometer.observation.Observation.observe(Observation.java:499)
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:124)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: io.camunda.common.exception.SdkException: Response not successful: 401
at io.camunda.common.http.DefaultHttpClient.parseAndRetry(DefaultHttpClient.java:295)
at io.camunda.common.http.DefaultHttpClient.post(DefaultHttpClient.java:153)
... 16 common frames omitted
Here is the full log after the helm upgrade before I restarted identity
connectors.yaml (176.6 KB)
and here is the part of the log after identity was restarted and connectors started up successfully
connectors_end.yaml (123.7 KB)