Background:
Im using self-managed Camunda 8.4.0 with only zeebe broker, gateway, operate and identity which connected to existing keycloak.
i turned on the multi tenancy feature with authentication.
I deployed workflow via the desktop modeler successfully.
Now when i upload worker for task type in Python with grpc-zeebe or in TypeScript with camunda8/sdk , i create channel/connection to the gateway i can assign to only one task type.
When i tried to assign to 2 different tasks to get their ActiveJobs (in python) or creating 2 workers in typescript, Im getting the error UNAUTHENTICATED:“Expected Identity to provide authorized tenants, see cause for details”
Things i tried:
Use different tokens for each worker
Use different connection to the gateway in same script and then different token
And still these two things didn’t went well and cause the same issue.
Hey thanks for quick respond.
This is the code in TypeScript. When I run, the 2 workers are ready, but after a while one of them fails with the error UNAUTHENTICATED:“Expected Identity to provide authorized tenants, see cause for details” and then it goes up again repeatedly.
The configuration of the identity:
One tenantId <default> which assigns to Zeebe application for the authorization, and still the issue occurde.
Im not getting any error when i connect to the gateway and print the topology, and i can deploy workflows, so the authentication works (i guess).
import { ZeebeGrpcClinet } from "@camunda8/sdk/dist/zeebe";
// create zeebe client with authentication
const zbc = new ZeebeGrpcClient({
config: {
ZEEBE_ADDRESS: "<gateway_endpoint>",
ZEEBE_CLIENT_ID: "zeebe",
ZEEBE_CLIENT_SECRET: "<secret>",
CAMUNDA_ZEEBE_OAUTH_AUDIENCE: "zeebe-api",
CAMUNDA_TENANT_ID: "<default>",
CAMUNDA_AUTH_STRATEGY: "OAUTH",
CAMUNDA_OAUTH_URL: "http://<keycloak_endpoint>/realms/camunda-platform/protocol/openid-connect/token",
CAMUNDA_SECURE_CONNECTION: false,
},
});
// workers demo handler
function handler(job) {
return job.complete({})
}
// create first worker
const zbwOne = zbc.createWorker({
taskType: "demo-task1",
taskHandler: handler,
})
// create second worker
const zbwTwo = zbc.createWorker({
taskType: "demo-task2",
taskHandler: handler,
})
// start first worker
zbwOne.on("ready", () => {
zbwOne.log("zeebe worker1 is ready!");
})
// start second worker
zbwTwo.on("ready", () => {
zbwOne.log("zeebe worker2 is ready!");
})
@IdanS - I don’t think I’ve seen this behavior before and couldn’t immediately replicate it. Does it still happen if you separate the workers into two separate scripts and launch then independently?
If I separate the workers into two scripts and launch them separately, it will work.
However, if I were to do this for 10 different task types, I would need 10 scripts, which is inefficient. Turning off multi-tenancy would work perfectly with this behavior.
Therefore, if I need to upgrade for multi-tenancy, it would require me to separate all the work I have done so far, which would be detrimental.
There is no way to allow for starting more than one worker in the same script on the same client connection?
@IdanS - I’m not suggesting separate scripts as a solution, it’s a troubleshooting step. It sounds like there may be an issue within the JavaScript client. I would suggest opening an issue in the GitHub repository for this, so the engineers can take a look!
There is no need to raise an issue for them because I believe it is a cluster problem. I am experiencing the same issue with the zeebe-grpc library in my Python script.
Could it be that Camunda 8.4.0 is not stable, or is there a potential issue with the communication between the gateway and identity components?
@IdanS - 8.4 is stable. However, zeebe-grpc may not be, as it’s a community maintained package that hasn’t had any updates in 6+ months. It’s still possible that it’s something with the library (or libraries). From what you’ve shared, it doesn’t seem like a cluster issue to me, because the job workers work as independent scripts. That particular error message doesn’t necessarily mean there’s a communication issue between Zeebe and Identity.
How long do you think it will take for me to receive a response from the developers and for the problem to be resolved?
I am dependent on zeebe-grpc due to using both Python and Typescript, so I may not be able to upgrade to multi-tenancy even if this issue be resolved in the Camunda8 SDK library?
@IdanS - I can’t say for certain. The Javascript/Typescript library is officially supported and maintained by Camunda, so if there is a bug, it will be something Camunda works to resolve. zeebe-grpc and related Python libraries (like pyzeebe) are maintained by the community and are dependent on other factors. If you need priority support you can contact our enterprise support team.
I have asked internally about the issue to see if anyone has encountered this before.
One other thing you could try, to further narrow the scope of the issue, is to use our docker-compose platform with the latest production version of Camunda (8.5) and see if you still have issues. If you don’t, you can then set up another local environment with 8.4 to see if there are issues. If there aren’t, then it may be an issue with your cluster; if there are, then it is more likely an issue with
@IdanS - after speaking with an engineer, turns out this is a known error that occurs when Zeebe doesn’t have enough resources to handle the requests. There is a GitHub issue, but it’s closed, so that’s why I didn’t find it originally (silly me!). Have a look at this issue, specifically this comment:
Hey, I noticed this issue before I opened it. I am using chart version 9.0.3, so I do not have the same key to configure. I changed the ioThreadsCount (which is configured on the Zeebe broker, not the gateway) and increased the resources to 2.5 CPUs and 5 threads.
However, I am still encountering the same error.