Not able to login in cluster mode

Hi,

We recently deployed 2 nodes cluster in Azure cloud but after deployment we are not able to login in console:

Getting below exception:

base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:189)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:892)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1038)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:908)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:712)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:461)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:384)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:312)
at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:394)
at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:253)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:175)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:834)
2022-05-31 09:22:33,265 [DEBUG] [053-exec-2] me.alidg.errors.WebErrorHandlers : The ‘me.alidg.errors.handlers.AnnotatedWebErrorHandler’ is going to handle the ‘me.alidg.errors.adapter.attributes.Exceptions$ForbiddenException’ exception
2022-05-31 09:22:33,272 [DEBUG] [053-exec-2] me.alidg.errors.WebErrorHandlers : Generated fingerprint: 08d51daf-68c6-4e9c-a4d5-243fcf7ad3ca
2022-05-31 09:22:33,277 [DEBUG] [053-exec-2] me.alidg.errors.WebErrorHandlers : About to execute 1 error handler post processors
2022-05-31 09:22:33,282 [ERROR] [053-exec-2] d.t.b.s.r.e.LoggingErrorHandlerProcessor: ERROR: HttpError{errors=[CodedMessage{code=‘security.access_denied’, message=‘Access Denied’, arguments=[]}], httpStatus=403 FORBIDDEN, fingerprint=08d51daf-68c6-4e9c-a4d5-243fcf7ad3ca}
2022-05-31 09:22:33,288 [WARN ] [053-exec-2] d.t.b.s.r.e.LoggingErrorHandlerProcessor: REST API Response Error → [Fingerprint: 08d51daf-68c6-4e9c-a4d5-243fcf7ad3ca]
Status: 403 FORBIDDEN

Getting below screen on UI.

Screenshot-2022-05-31-at-11-47-28-AM

Any help would be appreciated.

Can you explain a little more about what exactly happens when you try to log in?

It fails to login with that RED error message in UI.

Login Failed : Wrong credentials, locked user or missing access rights to application

Are you using a load balancer?

No, we aren’t using LoadBalancer, using ClusterIP on service to expose the pods.

If this only happens when you have more than one node, it’s likely because the first request to login goes to node 1 and after a successful login the next request goes to node 2 which does not know you’ve successfully logged in.

To prevent this you need to to make sure that when accessing the webapps all requests after the first keep the same session and speak to the same node.

So do we need to go for Load Balancer with Sticky sessions or Session manager like memcached ? Either of the two will work or both we need to implement ?

A load balancer with sticky sessions is the most common way to solve the problem (if indeed this is the issue you’re having) but there might be other solutions out there that I myself haven’t tried.