Camunda 8.3 After upgrade started to fail

Camunda 8 Topics
,
1

Hi Team,

I’ve been using Camunda for a while now, and all of my workflows were well with both self-managed and Camunda cloud. However, since updating to Camuda 8.3, none of my processes are working properly.

In my application, I’m getting the following warning messages.

2023-11-02 11:03:56.266 WARN 4839 — [lt-executor-518] io.camunda.zeebe.client.job.poller : Failed to activate jobs for worker updateRule and job type updateRule

io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
at io.grpc.Status.asRuntimeException(Status.java:539) ~[grpc-api-1.54.2.jar:1.54.2]
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:487) ~[grpc-stub-1.54.2.jar:1.54.2]
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:576) ~[grpc-core-1.54.2.jar:1.54.2]
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70) ~[grpc-core-1.54.2.jar:1.54.2]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:757) ~[grpc-core-1.54.2.jar:1.54.2]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:736) ~[grpc-core-1.54.2.jar:1.54.2]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.54.2.jar:1.54.2]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[grpc-core-1.54.2.jar:1.54.2]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:840) ~[na:na]
Caused by: io.netty.channel.ConnectTimeoutException: connection timed out: /127.0.0.1:26500
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:261) ~[netty-transport-4.1.84.Final.jar:4.1.84.Final]
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) ~[netty-common-4.1.85.Final.jar:4.1.85.Final]
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:153) ~[netty-common-4.1.85.Final.jar:4.1.85.Final]
at io.netty.util.concurrent.AbstractEventExecutor.runTask$$$capture(AbstractEventExecutor.java:174) ~[netty-common-4.1.85.Final.jar:4.1.85.Final]
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java) ~[netty-common-4.1.85.Final.jar:4.1.85.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:167) ~[netty-common-4.1.85.Final.jar:4.1.85.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java) ~[netty-common-4.1.85.Final.jar:4.1.85.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.85.Final.jar:4.1.85.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) ~[netty-transport-4.1.84.Final.jar:4.1.84.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.85.Final.jar:4.1.85.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.85.Final.jar:4.1.85.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.85.Final.jar:4.1.85.Final]
… 1 common frames omitted

Environment setup:

  • Machine : Mac M1 Pro
  • Java version : 17
  • Spring boot version: 2.7.5
  • camunda version : 8.3.0

Can some one please suggest the way to solve this problem ?

Regards,
Raja Gandharaw.

2

Hi @GRajaMca - is this happening when your job workers are trying to connect to a local self-managed installation or SaaS or both?

3

Hi @nathan.loding , its happening for both.

Note: After lots of attempt figured out that this issue is inconsistency one.

4

What’s the cause for inconsistency? Versions of camunda components or configuration issue in charts?

5

@GRajaMca - what version of spring-zeebe / zeebe-client are you running? I’m not aware of any incompatibilities, but might be good to first make sure you are using the latest version.

You also said it was inconsistent. If you restart your application after you get the error, does it work correctly, or does it sometimes take a few tries? One other thing you can try is to use zbctl to try to connect to Zeebe when the Spring Boot startup fails. This would help eliminate your application as the issue and would point to something else (perhaps a networking issue of some kind?).

1 Like
6

@nathan.loding , @aravindhrs ,

We concluded that this problem is caused by some internal network configuration difficulties from the VPN after extensive testing.

Everything worked perfectly when I tried to reload everything on my local machine.

1 Like
7

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.