SpringSecurity Authentication is null when calling camundaClient.newCreateInstanceCommand()

I am using camunda-spring-boot-starter 8.8.2 with c8run 8.8.2

and in my application.properties I have disabled camundaClient as follows:



camunda.client.enabled=false
camunda.client.grpc-address=http://localhost:26500
camunda.client.rest-address=http://localhost:8080
camunda.client.prefer-rest-over-grpc=false

and my CamundaClientConfig is as follows:

@Configuration
public class CamundaClientConfig {

@Value("${camunda.client.rest-address}")
private String camundaRestAddress;

@Value("${camunda.client.grpc-address}")
private String camundaGrpcAddress;

@Value("${camunda.client.prefer-rest-over-grpc}")
private boolean preferRestOverGrpc;

@Bean
public CamundaClient camundaClient() {

	CamundaClientBuilder builder = CamundaClient.newClientBuilder();
	builder.credentialsProvider(new BearerTokenCredentialsProvider());
	builder.restAddress(URI.create(camundaRestAddress));
	builder.grpcAddress(URI.create(camundaGrpcAddress));
	builder.preferRestOverGrpc(preferRestOverGrpc);
	return builder.build();
}
}

my BearerTokenCredentialsProvider:

public class BearerTokenCredentialsProvider implements CredentialsProvider {
public BearerTokenCredentialsProvider() {

}

@Override
public void applyCredentials(final CredentialsApplier applier) {
	applier.put("Authorization", SecurityUtils.getBearerToken());
}

@Override
public boolean shouldRetryRequest(final StatusCode status) {
	return status.isUnauthorized();
}

}

the getBearerToken method:

public static String getBearerToken() {
  var authentication = SecurityContextHolder.getContext().getAuthentication();
  if (authentication instanceof JwtAuthenticationToken jwtAuth) {
    String bearerToken = jwtAuth.getToken().getTokenValue();
    return "Bearer " + bearerToken;
  }
  return null;
}

my createProcess instance API:

@PostMapping("/process-instances/create")	
@PreAuthorize("isAuthenticated()")
ProcessInstanceEvent createProcessInstance(@RequestBody CreateProcessInstanceRequest createProcessInstanceRequest) {
   return camundaService.createProcessInstance(createProcessInstanceRequest);

}

my camundaService.createProcessInstance:

public ProcessInstanceEvent createProcessInstance(CreateProcessInstanceRequest request) {
if (request.getVariablesMap() == null) {
    request.setVariablesMap(new HashMap<String, Object>());
}
request.getVariablesMap().put(CamundaVariables.CREATOR, SecurityUtils.getUserNameFromAccessToken());
request.getVariablesMap().put(CamundaVariables.REQUEST_STATUS_NAME, "DRAFT");
// bearerToken here is correct and NOT NULL
log.info("####### bearerToken before newCreateInstanceCommand(): " + SecurityUtils.getBearerToken());
// implicit call here for BearerTokenCredentialsProvider where
// SecurityUtils.getBearerToken() will be NULL 
ProcessInstanceEvent processInstanceEvent = camundaClient.newCreateInstanceCommand().bpmnProcessId(request.getBpmnProcessId()).latestVersion().variables(request.getVariablesMap()).send().join();
return processInstanceEvent;

}

Please advise why the bearer token is NULL inside BearerTokenCredentialsProvider

NOTE: using other camundaClient methods works fine witn no issues like:

camundaClient.newCompleteUserTaskCommand(taskId)

Hi @devmsaleh,

This is a common issue when using Spring Security with the Camunda client in async operations. The problem you’re experiencing is related to thread context propagation.

Root Cause

The SecurityContextHolder in Spring Security is thread-local by default. When you call camundaClient.newCreateInstanceCommand().send().join(), the Camunda client executes the operation asynchronously in a different thread. In this new thread, the Spring Security context is not available, which is why SecurityUtils.getBearerToken() returns null inside your BearerTokenCredentialsProvider.

Solutions

Here are a few approaches to solve this:

Option 1: Capture the Token Before the Async Call

Modify your service method to capture the bearer token before making the async call:

public ProcessInstanceEvent createProcessInstance(CreateProcessInstanceRequest request) {
    if (request.getVariablesMap() == null) {
        request.setVariablesMap(new HashMap<String, Object>());
    }
    request.getVariablesMap().put(CamundaVariables.CREATOR, SecurityUtils.getUserNameFromAccessToken());
    request.getVariablesMap().put(CamundaVariables.REQUEST_STATUS_NAME, "DRAFT");
    
    // Capture the bearer token in the current thread
    String bearerToken = SecurityUtils.getBearerToken();
    
    // Create a client with the captured token
    CamundaClient clientWithToken = CamundaClient.newClientBuilder()
        .credentialsProvider(new StaticBearerTokenCredentialsProvider(bearerToken))
        .restAddress(URI.create(camundaRestAddress))
        .grpcAddress(URI.create(camundaGrpcAddress))
        .preferRestOverGrpc(preferRestOverGrpc)
        .build();
    
    ProcessInstanceEvent processInstanceEvent = clientWithToken.newCreateInstanceCommand()
        .bpmnProcessId(request.getBpmnProcessId())
        .latestVersion()
        .variables(request.getVariablesMap())
        .send()
        .join();
        
    return processInstanceEvent;
}

And create a static credentials provider:

public class StaticBearerTokenCredentialsProvider implements CredentialsProvider {
    private final String bearerToken;
    
    public StaticBearerTokenCredentialsProvider(String bearerToken) {
        this.bearerToken = bearerToken;
    }
    
    @Override
    public void applyCredentials(final CredentialsApplier applier) {
        applier.put("Authorization", bearerToken);
    }
    
    @Override
    public boolean shouldRetryRequest(final StatusCode status) {
        return status.isUnauthorized();
    }
}

Option 2: Configure Security Context Propagation

Configure Spring Security to propagate the security context to async threads by adding this to your configuration:

@Configuration
@EnableAsync
public class AsyncConfig implements AsyncConfigurer {
    
    @Override
    public Executor getAsyncExecutor() {
        ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
        executor.setCorePoolSize(5);
        executor.setMaxPoolSize(10);
        executor.setQueueCapacity(100);
        executor.setThreadNamePrefix("async-");
        executor.initialize();
        return new DelegatingSecurityContextAsyncTaskExecutor(executor);
    }
}

Option 3: Manual Context Propagation

Manually propagate the security context:

public ProcessInstanceEvent createProcessInstance(CreateProcessInstanceRequest request) {
    // ... your existing code ...
    
    // Capture the current security context
    SecurityContext securityContext = SecurityContextHolder.getContext();
    
    CompletableFuture<ProcessInstanceEvent> future = CompletableFuture.supplyAsync(() -> {
        try {
            // Set the security context in the new thread
            SecurityContextHolder.setContext(securityContext);
            
            return camundaClient.newCreateInstanceCommand()
                .bpmnProcessId(request.getBpmnProcessId())
                .latestVersion()
                .variables(request.getVariablesMap())
                .send()
                .join();
        } finally {
            // Clear the context after use
            SecurityContextHolder.clearContext();
        }
    });
    
    return future.join();
}

Recommendation

I’d recommend Option 1 as it’s the most straightforward and doesn’t require complex async configuration. It also ensures that each request uses the correct token without relying on thread-local storage.

The issue you’re experiencing is well-documented in Spring Security when dealing with async operations. The key is to capture the authentication details in the original request thread before the async operation begins.

References:

Let me know if you need any clarification or if you’d like help implementing any of these solutions!

So I tried option 2 and It didn’t work

And I tried option 1 as per your recommendation, but sometimes it produces the following error:

`2025-11-24T13:41:31.303+03:00 ERROR 840 — [camunda-api] [io-7777-exec-10] i.g.i.ManagedChannelOrphanWrapper        : `*`~`*`~`*`~ Previous channel ManagedChannelImpl{logId=3, target=localhost:26500} was garbage collected without being shut down! ~`*`~`*`~`*
`Make sure to call shutdown()/shutdownNow()`
`java.lang.RuntimeException: ManagedChannel allocation site`
`at io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.(ManagedChannelOrphanWrapper.java:102 undefined) ~[grpc-core-1.73.0.jar:1.73.0]`
`at io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:60 undefined) ~[grpc-core-1.73.0.jar:1.73.0]`
`at io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:51 undefined) ~[grpc-core-1.73.0.jar:1.73.0]`
`at io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:731 undefined) ~[grpc-core-1.73.0.jar:1.73.0]`
`at io.grpc.ForwardingChannelBuilder2.build(ForwardingChannelBuilder2.java:278 undefined) ~[grpc-api-1.73.0.jar:1.73.0]`
`at io.camunda.client.impl.CamundaClientImpl.buildChannel(CamundaClientImpl.java:418 undefined) ~[camunda-client-java-8.8.2.jar:8.8.2]`
`at io.camunda.client.impl.CamundaClientImpl.(CamundaClientImpl.java:336 undefined) ~[camunda-client-java-8.8.2.jar:8.8.2]`
`at io.camunda.client.impl.CamundaClientBuilderImpl.build(CamundaClientBuilderImpl.java:582 undefined) ~[camunda-client-java-8.8.2.jar:8.8.2]`
`at sa.gov.ncgr.service.CamundaService.createProcessInstance(CamundaService.java:184 undefined) ~[classes/:na]`
`at sa.gov.ncgr.controller.CamundaController.createProcessInstance(CamundaController.java:62 undefined) ~[classes/:na]`
`at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103 undefined) ~[na:na]`
`at java.base/java.lang.reflect.Method.invoke(Method.java:580 undefined) ~[na:na]`
`at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:360 undefined) ~[spring-aop-6.2.11.jar:6.2.11]`
`at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196 undefined) ~[spring-aop-6.2.11.jar:6.2.11]`
`at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163 undefined) ~[spring-aop-6.2.11.jar:6.2.11]`
`at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:174 undefined) ~[spring-context-6.2.11.jar:6.2.11]`
`at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184 undefined) ~[spring-aop-6.2.11.jar:6.2.11]`
`at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:67 undefined) ~[spring-security-core-6.5.5.jar:6.5.5]`
`at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184 undefined) ~[spring-aop-6.2.11.jar:6.2.11]`
`at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:728 undefined) ~[spring-aop-6.2.11.jar:6.2.11]`
`at sa.gov.ncgr.controller.CamundaController$SpringCGLIB$0.createProcessInstance( undefined) ~[classes/:na]`
`at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103 undefined) ~[na:na]`
`at java.base/java.lang.reflect.Method.invoke(Method.java:580 undefined) ~[na:na]`
`at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:258 undefined) ~[spring-web-6.2.11.jar:6.2.11]`
`at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:191 undefined) ~[spring-web-6.2.11.jar:6.2.11]`
`at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118 undefined) ~[spring-webmvc-6.2.11.jar:6.2.11]`
`at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:991 undefined) ~[spring-webmvc-6.2.11.jar:6.2.11]`
`at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:896 undefined) ~[spring-webmvc-6.2.11.jar:6.2.11]`
`at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87 undefined) ~[spring-webmvc-6.2.11.jar:6.2.11]`
`at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1089 undefined) ~[spring-webmvc-6.2.11.jar:6.2.11]`
`at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:979 undefined) ~[spring-webmvc-6.2.11.jar:6.2.11]`
`at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014 undefined) ~[spring-webmvc-6.2.11.jar:6.2.11]`
`at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:914 undefined) ~[spring-webmvc-6.2.11.jar:6.2.11]`
`at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:590 undefined) ~[tomcat-embed-core-10.1.46.jar:6.0]`
`at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885 undefined) ~[spring-webmvc-6.2.11.jar:6.2.11]`
`at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658 undefined) ~[tomcat-embed-core-10.1.46.jar:6.0]`
`at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:195 undefined) ~[tomcat-embed-core-10.1.46.jar:10.1.46]`
`at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140 undefined) ~[tomcat-embed-core-10.1.46.jar:10.1.46]`
`at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51 undefined) ~[tomcat-embed-websocket-10.1.46.jar:10.1.46]`
`at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164 undefined) ~[tomcat-embed-core-10.1.46.jar:10.1.46]`
`at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140 undefined) ~[tomcat-embed-core-10.1.46.jar:10.1.46]`
`at org.springframework.web.servlet.resource.ResourceUrlEncodingFilter.doFilter(ResourceUrlEncodingFilter.java:66 undefined) ~[spring-webmvc-6.2.11.jar:6.2.11]`
`at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164 undefined) ~[tomcat-embed-core-10.1.46.jar:10.1.46]`
`at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140 undefined) ~[tomcat-embed-core-10.1.46.jar:10.1.46]`
`at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:108 undefined) ~[spring-web-6.2.11.jar:6.2.11]`
`at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:108 undefined) ~[spring-web-6.2.11.jar:6.2.11]`
`at org.springframework.security.web.FilterChainProxy.lambda$doFilterInternal$3(FilterChainProxy.java:231 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:365 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:117 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:125 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:179 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.security.web.authentication.AuthenticationFilter.doFilterInternal(AuthenticationFilter.java:177 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116 undefined) ~[spring-web-6.2.11.jar:6.2.11]`
`at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.security.oauth2.server.resource.web.authentication.BearerTokenAuthenticationFilter.doFilterInternal(BearerTokenAuthenticationFilter.java:158 undefined) ~[spring-security-oauth2-resource-server-6.5.5.jar:6.5.5]`
`at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116 undefined) ~[spring-web-6.2.11.jar:6.2.11]`
`at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:107 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:93 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116 undefined) ~[spring-web-6.2.11.jar:6.2.11]`
`at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:82 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:69 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116 undefined) ~[spring-web-6.2.11.jar:6.2.11]`
`at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116 undefined) ~[spring-web-6.2.11.jar:6.2.11]`
`at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:233 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:191 undefined) ~[spring-security-web-6.5.5.jar:6.5.5]`
`at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113 undefined) ~[spring-web-6.2.11.jar:6.2.11]`
`at org.springframework.web.filter.ServletRequestPathFilter.doFilter(ServletRequestPathFilter.java:52 undefined) ~[spring-web-6.2.11.jar:6.2.11]`
`at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113 undefined) ~[spring-web-6.2.11.jar:6.2.11]`
`at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:74 undefined) ~[spring-web-6.2.11.jar:6.2.11]`
`at org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration$CompositeFilterChainProxy.doFilter(WebSecurityConfiguration.java:319 undefined) ~[spring-security-config-6.5.5.jar:6.5.5]`
`at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113 undefined) ~[spring-web-6.2.11.jar:6.2.11]`
`at org.springframework.web.servlet.handler.HandlerMappingIntrospector.lambda$createCacheFilter$4(HandlerMappingIntrospector.java:267 undefined) ~[spring-webmvc-6.2.11.jar:6.2.11]`
`at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113 undefined) ~[spring-web-6.2.11.jar:6.2.11]`
`at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:74 undefined) ~[spring-web-6.2.11.jar:6.2.11]`
`at org.springframework.security.config.annotation.web.configuration.WebMvcSecurityConfiguration$CompositeFilterChainProxy.doFilter(WebMvcSecurityConfiguration.java:240 undefined) ~[spring-security-config-6.5.5.jar:6.5.5]`
`at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:362 undefined) ~[spring-web-6.2.11.jar:6.2.11]`
`at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:278 undefined) ~[spring-web-6.2.11.jar:6.2.11]`
`at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164 undefined) ~[tomcat-embed-core-10.1.46.jar:10.1.46]`
`at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140 undefined) ~[tomcat-embed-core-10.1.46.jar:10.1.46]`
`at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100 undefined) ~[spring-web-6.2.11.jar:6.2.11]`
`at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116 undefined) ~[spring-web-6.2.11.jar:6.2.11]`
`at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164 undefined) ~[tomcat-embed-core-10.1.46.jar:10.1.46]`
`at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140 undefined) ~[tomcat-embed-core-10.1.46.jar:10.1.46]`
`at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93 undefined) ~[spring-web-6.2.11.jar:6.2.11]`
`at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116 undefined) ~[spring-web-6.2.11.jar:6.2.11]`
`at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164 undefined) ~[tomcat-embed-core-10.1.46.jar:10.1.46]`
`at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140 undefined) ~[tomcat-embed-core-10.1.46.jar:10.1.46]`
`at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201 undefined) ~[spring-web-6.2.11.jar:6.2.11]`
`at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116 undefined) ~[spring-web-6.2.11.jar:6.2.11]`
`at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164 undefined) ~[tomcat-embed-core-10.1.46.jar:10.1.46]`
`at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140 undefined) ~[tomcat-embed-core-10.1.46.jar:10.1.46]`
`at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167 undefined) ~[tomcat-embed-core-10.1.46.jar:10.1.46]`
`at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90 undefined) ~[tomcat-embed-core-10.1.46.jar:10.1.46]`
`at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:483 undefined) ~[tomcat-embed-core-10.1.46.jar:10.1.46]`
`at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:116 undefined) ~[tomcat-embed-core-10.1.46.jar:10.1.46]`
`at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93 undefined) ~[tomcat-embed-core-10.1.46.jar:10.1.46]`
`at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74 undefined) ~[tomcat-embed-core-10.1.46.jar:10.1.46]`
`at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344 undefined) ~[tomcat-embed-core-10.1.46.jar:10.1.46]`
`at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:398 undefined) ~[tomcat-embed-core-10.1.46.jar:10.1.46]`
`at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63 undefined) ~[tomcat-embed-core-10.1.46.jar:10.1.46]`
`at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:903 undefined) ~[tomcat-embed-core-10.1.46.jar:10.1.46]`
`at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1776 undefined) ~[tomcat-embed-core-10.1.46.jar:10.1.46]`
`at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52 undefined) ~[tomcat-embed-core-10.1.46.jar:10.1.46]`
`at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:975 undefined) ~[tomcat-embed-core-10.1.46.jar:10.1.46]`
`at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:493 undefined) ~[tomcat-embed-core-10.1.46.jar:10.1.46]`
`at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63 undefined) ~[tomcat-embed-core-10.1.46.jar:10.1.46]`
`at java.base/java.lang.Thread.run(Thread.java:1583 undefined) ~[na:na]`

I see the issue! The error you’re getting is a resource leak - you’re creating new CamundaClient instances on every request but not properly closing them. This causes gRPC channels to accumulate and eventually get garbage collected without being properly shut down.

The Problem

In Option 1, you’re creating a new CamundaClient for each request:

CamundaClient clientWithToken = CamundaClient.newClientBuilder()
    .credentialsProvider(new StaticBearerTokenCredentialsProvider(bearerToken))
    // ... other config
    .build();

Each CamundaClient creates its own gRPC channel, and if you don’t explicitly close it, you’ll get resource leaks.

Better Solution: Client Pool or Singleton with Dynamic Credentials

Here’s a more efficient approach that reuses the client but dynamically provides credentials:

Option A: Modified Credentials Provider (Recommended)

Keep your original single CamundaClient bean but modify your credentials provider to capture the token at the time of the request:

@Configuration
public class CamundaClientConfig {

    @Value("${camunda.client.rest-address}")
    private String camundaRestAddress;

    @Value("${camunda.client.grpc-address}")
    private String camundaGrpcAddress;

    @Value("${camunda.client.prefer-rest-over-grpc}")
    private boolean preferRestOverGrpc;

    @Bean
    public CamundaClient camundaClient() {
        CamundaClientBuilder builder = CamundaClient.newClientBuilder();
        builder.credentialsProvider(new DynamicBearerTokenCredentialsProvider());
        builder.restAddress(URI.create(camundaRestAddress));
        builder.grpcAddress(URI.create(camundaGrpcAddress));
        builder.preferRestOverGrpc(preferRestOverGrpc);
        return builder.build();
    }
}

And create a new credentials provider that captures the token when needed:

public class DynamicBearerTokenCredentialsProvider implements CredentialsProvider {
    
    private static final ThreadLocal<String> tokenHolder = new ThreadLocal<>();
    
    public static void setToken(String token) {
        tokenHolder.set(token);
    }
    
    public static void clearToken() {
        tokenHolder.remove();
    }
    
    @Override
    public void applyCredentials(final CredentialsApplier applier) {
        String token = tokenHolder.get();
        if (token != null) {
            applier.put("Authorization", token);
        } else {
            // Fallback - try to get from SecurityContext if available
            String fallbackToken = SecurityUtils.getBearerToken();
            if (fallbackToken != null) {
                applier.put("Authorization", fallbackToken);
            }
        }
    }
    
    @Override
    public boolean shouldRetryRequest(final StatusCode status) {
        return status.isUnauthorized();
    }
}

Then modify your service method:

public ProcessInstanceEvent createProcessInstance(CreateProcessInstanceRequest request) {
    if (request.getVariablesMap() == null) {
        request.setVariablesMap(new HashMap<String, Object>());
    }
    request.getVariablesMap().put(CamundaVariables.CREATOR, SecurityUtils.getUserNameFromAccessToken());
    request.getVariablesMap().put(CamundaVariables.REQUEST_STATUS_NAME, "DRAFT");
    
    // Capture the bearer token and set it in ThreadLocal
    String bearerToken = SecurityUtils.getBearerToken();
    DynamicBearerTokenCredentialsProvider.setToken(bearerToken);
    
    try {
        ProcessInstanceEvent processInstanceEvent = camundaClient.newCreateInstanceCommand()
            .bpmnProcessId(request.getBpmnProcessId())
            .latestVersion()
            .variables(request.getVariablesMap())
            .send()
            .join();
        
        return processInstanceEvent;
    } finally {
        // Always clear the token from ThreadLocal to prevent memory leaks
        DynamicBearerTokenCredentialsProvider.clearToken();
    }
}

Option B: Client Pool (Alternative)

If you prefer the multiple client approach, you need to properly manage the lifecycle:

@Service
public class CamundaService {
    
    @Value("${camunda.client.rest-address}")
    private String camundaRestAddress;
    
    @Value("${camunda.client.grpc-address}")
    private String camundaGrpcAddress;
    
    @Value("${camunda.client.prefer-rest-over-grpc}")
    private boolean preferRestOverGrpc;
    
    public ProcessInstanceEvent createProcessInstance(CreateProcessInstanceRequest request) {
        if (request.getVariablesMap() == null) {
            request.setVariablesMap(new HashMap<String, Object>());
        }
        request.getVariablesMap().put(CamundaVariables.CREATOR, SecurityUtils.getUserNameFromAccessToken());
        request.getVariablesMap().put(CamundaVariables.REQUEST_STATUS_NAME, "DRAFT");
        
        String bearerToken = SecurityUtils.getBearerToken();
        
        // Create client with proper resource management
        CamundaClient clientWithToken = CamundaClient.newClientBuilder()
            .credentialsProvider(new StaticBearerTokenCredentialsProvider(bearerToken))
            .restAddress(URI.create(camundaRestAddress))
            .grpcAddress(URI.create(camundaGrpcAddress))
            .preferRestOverGrpc(preferRestOverGrpc)
            .build();
        
        try {
            ProcessInstanceEvent processInstanceEvent = clientWithToken.newCreateInstanceCommand()
                .bpmnProcessId(request.getBpmnProcessId())
                .latestVersion()
                .variables(request.getVariablesMap())
                .send()
                .join();
            
            return processInstanceEvent;
        } finally {
            // IMPORTANT: Always close the client to prevent resource leaks
            clientWithToken.close();
        }
    }
}

Recommendation

I strongly recommend Option A (Dynamic Credentials Provider) because:

  1. Better Performance: Reuses the same client and gRPC connections
  2. No Resource Leaks: No need to manage client lifecycle
  3. Thread-Safe: Uses ThreadLocal to isolate tokens per request
  4. Cleaner Code: Less boilerplate and resource management

The key insight is that you don’t need multiple clients - you just need to dynamically provide the credentials at request time. The ThreadLocal approach ensures that each request thread has its own token context.

Try Option A and let me know if it resolves both the authentication and resource leak issues!

Unfortunately, I tried option A, and there is a weird behavior happening, that the applyCredentials is getting executed three times, and each time the token is null again, so option A for the dynamic token doesn’t work with some methods like newCreateInstanceCommand
so It seems that I will be using option 2 and close the token after each request
but my question is there are any disadvantaes of creating new camunda client for each request ?

You’re absolutely right to ask about the disadvantages! Creating a new CamundaClient for each request has several significant performance and resource implications:

Disadvantages of Creating New CamundaClient Per Request

1. Connection Overhead

  • Each CamundaClient creates new HTTP/gRPC connections to Camunda
  • Connection establishment has latency overhead (TCP handshake, TLS negotiation, etc.)
  • No connection pooling benefits - each request pays the full connection cost

2. Resource Consumption

  • Memory: Each client allocates internal buffers, thread pools, and connection resources
  • CPU: Connection establishment and teardown consume CPU cycles
  • File Descriptors: Each connection uses system file descriptors (can hit OS limits)

3. Performance Impact

According to Camunda’s performance investigations:

  • REST API calls are already more CPU-heavy than gRPC
  • Creating new connections repeatedly would further increase CPU overhead
  • At higher loads, this becomes a significant bottleneck

4. Missing Optimizations

  • No Keep-Alive: Fresh connections can’t benefit from HTTP keep-alive
  • No Connection Reuse: TCP connection warm-up benefits are lost
  • Internal Queues: Client internal optimization structures are recreated each time

Better Alternative: Request-Scoped Token Injection

Since the dynamic ThreadLocal approach had issues with multiple applyCredentials calls, here’s a more robust solution:

@Service
public class CamundaService {
    
    private final CamundaClient camundaClient;
    
    public CamundaService(CamundaClient camundaClient) {
        this.camundaClient = camundaClient;
    }
    
    public ProcessInstanceEvent createProcessInstance(CreateProcessInstanceRequest request) {
        if (request.getVariablesMap() == null) {
            request.setVariablesMap(new HashMap<String, Object>());
        }
        request.getVariablesMap().put(CamundaVariables.CREATOR, SecurityUtils.getUserNameFromAccessToken());
        request.getVariablesMap().put(CamundaVariables.REQUEST_STATUS_NAME, "DRAFT");
        
        // Capture token before any async operations
        String bearerToken = SecurityUtils.getBearerToken();
        
        // Use a wrapper that injects the token into the command
        return executeWithToken(bearerToken, () -> 
            camundaClient.newCreateInstanceCommand()
                .bpmnProcessId(request.getBpmnProcessId())
                .latestVersion()
                .variables(request.getVariablesMap())
                .send()
                .join()
        );
    }
    
    private <T> T executeWithToken(String token, Supplier<T> operation) {
        // Store current context
        SecurityContext originalContext = SecurityContextHolder.getContext();
        
        try {
            // Create a new context with the captured token
            SecurityContext newContext = SecurityContextHolder.createEmptyContext();
            JwtAuthenticationToken auth = createAuthTokenFromBearer(token);
            newContext.setAuthentication(auth);
            SecurityContextHolder.setContext(newContext);
            
            return operation.get();
        } finally {
            // Restore original context
            SecurityContextHolder.setContext(originalContext);
        }
    }
    
    private JwtAuthenticationToken createAuthTokenFromBearer(String bearerToken) {
        // Extract JWT from "Bearer " prefix
        String jwtToken = bearerToken.substring(7);
        // You'll need to decode/validate the JWT here based on your setup
        // This is a simplified example - adapt to your JWT handling
        Jwt jwt = jwtDecoder.decode(jwtToken); // Inject JwtDecoder
        return new JwtAuthenticationToken(jwt);
    }
}

If You Must Use Multiple Clients

If you absolutely need the multiple client approach, at least implement client pooling to reuse clients:

@Service
public class CamundaClientPool {
    private final Map<String, CamundaClient> clientPool = new ConcurrentHashMap<>();
    private final int maxPoolSize = 10; // Adjust based on your needs
    
    public CamundaClient getClientForToken(String token) {
        return clientPool.computeIfAbsent(token, this::createClient);
    }
    
    private CamundaClient createClient(String token) {
        if (clientPool.size() >= maxPoolSize) {
            // Implement eviction strategy (LRU, etc.)
            evictOldestClient();
        }
        
        return CamundaClient.newClientBuilder()
            .credentialsProvider(new StaticBearerTokenCredentialsProvider(token))
            .restAddress(URI.create(camundaRestAddress))
            .grpcAddress(URI.create(camundaGrpcAddress))
            .preferRestOverGrpc(preferRestOverGrpc)
            .build();
    }
    
    @PreDestroy
    public void cleanup() {
        clientPool.values().forEach(CamundaClient::close);
        clientPool.clear();
    }
}

Recommendation

The best practice according to Camunda’s documentation and performance research is to:

  1. Use a single, long-lived CamundaClient managed by Spring
  2. Handle authentication at the request level rather than client level
  3. Leverage connection pooling and keep-alive for optimal performance

Creating new clients per request works but comes with significant performance penalties that will become more apparent under load. The connection overhead alone can add 10-50ms per request depending on network conditions.

Would you like me to help you implement the request-scoped token injection approach? It should solve the authentication issue while maintaining optimal performance.

Unfortunately the Request-Scoped Token Injection didn’t work, still there’s a call to applyCredentials and the token is null there, and when tried to enable the default camundaClient also I got the error:

io.camunda.client.api.command.ClientStatusException: Expected authentication information at header with key [authorization], but found nothing

so @zee-bot can you please assign this issue to development team to support me on this

Hi @devmsaleh,

I understand the frustration with this authentication issue. You’ve tried multiple approaches and the core problem persists - the applyCredentials method in your BearerTokenCredentialsProvider is receiving a null token during async operations.

This appears to be a complex interaction between Spring Security context propagation and the Camunda client’s async execution model that requires deeper investigation from our development team.

I’m escalating this to a Camunda expert who can:

  1. Investigate the specific behavior of newCreateInstanceCommand() and how it handles credentials providers
  2. Provide guidance on the proper way to handle Spring Security context with the Camunda client in async scenarios
  3. Potentially identify if this is a known issue or limitation that needs to be addressed

A Camunda expert will follow up with you shortly to help resolve this authentication challenge.

Thank you for your patience and for providing such detailed information about your setup and the various approaches you’ve tried!

Hi @devmsaleh - this isn’t so much a Camunda question as it is a Spring/Java question. I am not a Java expert, but it looks like the bots first answer was the correct approach to solve the thread-local issue with the Spring security context. The new error produced is because the client connection isn’t being closed each time, and isn’t related the to the bearer token.

Have you tried a singleton pattern for the client (in combination with the StaticBearerTokenCredentialsProvider idea)? That would be the next thing I would try.

Also of note, this isn’t a Camunda support forum, this is a community forum; if you are needing faster support, or are looking for support from Camunda’s engineers and support staff, you need to open a support ticket.

Hi, thanks for your answer. I found out that the issue happens only when using grpc connections.

So when camunda.client.prefer-rest-over-grpc=true
This issue doesn’t appear at all.

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.