I am encountering numerous DeadlineExceed errors when attempting to set variables using the ZeebeClient C# SDK, even though the CPU and memory of the Zeebe Cluster and Gateway are performing slowly. The Camunda instance is deployed on AKS. The source code used to set variables is as follows:
public async Task SetVariablesAsync(long instanceKey, params KeyValuePair<string, object>[] variables)
{
var camundaVariables = new CamundaVariables(variables);
await _zeebeClient
.NewSetVariablesCommand(instanceKey)
.Variables(JsonSerializer.Serialize(variables))
.SendWithRetry(_retryTimeout);
}
Stack Trace
Exception: System.AggregateException: One or more errors occurred. (Status(StatusCode="DeadlineExceeded", Detail=""))
---> Grpc.Core.RpcException: Status(StatusCode="DeadlineExceeded", Detail="")
at Zeebe.Client.Impl.Commands.SetVariablesCommand.Send(Nullable`1 timeout, CancellationToken token)
at Zeebe.Client.Impl.Misc.TransientGrpcErrorRetryStrategy.DoWithRetry[TResult](Func`1 action)
at Zeebe.Client.Impl.Misc.TransientGrpcErrorRetryStrategy.DoWithRetry[TResult](Func`1 action)
at Zeebe.Client.Impl.Commands.SetVariablesCommand.SendWithRetry(Nullable`1 timespan, CancellationToken token)
To add some more context, we’re using clients from AWS to Camunda Cloud - and from local debug from Visuaal Studio to Camunda Cloud. All connections seem to be affected by the same problem using zb_client 2.2.0. Downgrading to 1.3.0 fixes the problem.
Hi @Larry and @daghb (welcome to the forums!) - if I’m reading this correctly, it sounds like an issue with the C# client, is that correct? If you use a different tool, such as zbctl, does it work as expected?
Hi @nathan.loding, I’m not certain if the issue is specific to the C# client, but yes, I am using the C# client. I haven’t tried a different tool yet. I believe I’m on the same page as @daghb when the issue occurred after upgrading to 2.2.0, I will downgrade to 1.3.0 and try again.
To me, it is indeed related to v2.2.0 of the client. I run the exact same code in 1.3.0 with no issue. We’ve had a lot of “Deadline exceeded” and learnt to cope with those - retrying after timeouts, but this one comes immediately. My favourite case is publishing two workflows to Camunda Cloud (8.2.16).
The exception happens immediately after calling Deploy, no timeout is functional.
Thanks for the information @daghb, @aravindhrs, and @Larry … do any of you have an enterprise plan with Camunda currently? If you do, I would suggest you also submit a support ticket, as that will escalate the priority of this issue.
Yes, we do. But anyway, the error seems to be that the client is not compatible with .NET 8, even if .NET 6 libraries should be backward compatible, according to Microsoft. Until the final release of .NET 8 arrives November 14, we’ll just fork the source, build a client using the .NET 8 SDK and this seems to work nicely without code changes. After Nov 14, Christopher will look into it and eventually build a .NET 8 version of the zb_client that we can use without having to build our own.
Hi @daghb ,
Have you tried using SDK 2.2.0 with .NET 6? In my case, I encountered an error when using .NET 6 with SDK 2.2.0.
I attempted to limit the active jobs and worker I/O through configuration, but I’m still experiencing the error. Have you had a similar experience?