Hi,
we have a sealed environment and need to use delegated SAS Token. For Operate and tasklist hte backup over Elasticsearch snapshot worked good. But with the Zeebe i got an Authorizationfailure.
<?xml version=\"1.0\" encoding=\"utf-8\"?>AuthorizationFailureThis request is not authorized to perform this operation.\nRequestId:a611b2b9-101e-006a-6f41-00451e000000\nTime:2024-09-06T09:46:45.6524292Z
its the same sas token as for operate and tasklist.
Configuration is by environment variables:
Helm Chart:
zeebe:
clusterSize: 1
partitionCount: 1
replicationFactor: 1
pvcSize: 10Gi
existingSecret: “secret”
env:
- name: ZEEBE_BROKER_DATA_BACKUP_STORE
value: AZURE
- name: ZEEBE_BROKER_DATA_BACKUP_AZURE_CONNECTIONSTRING
value: “BlobEndpoint=https://blobstorageaccountname.blob.core.windows.net;SharedAccessSignature=skoid=guid&sktid=guid&skt=2024-09-06T07%3A54%3A23Z&ske=2024-09-08T22%3A00%3A00Z&sks=b&skv=2021-04-10&sv=2021-04-10&se=2024-09-08T22%3A00%3A00Z&sr=c&sp=racwdlf&sig=sig”
- name: ZEEBE_BROKER_DATA_BACKUP_AZURE_BASEPATH
value: blobstorecontainer
Some questions for that:
we only permissioned the blobstorecontainer with Contributor, Storage Blob Data Contributor, Storage Blob Data Owner, Storage Blob Delegator (but this is more for the delegated SAS Token) what would be the needed Role permissions for the backup
i wanted to use the same blobcontainer for elastic search and zeebe, could that be a problem
delegated SAS Token only have a max lifetime of some days. Due to that we will rotate the key with every backup. For ElasticSearch that worked we simply changed the secret keys and then reloaded them.
For Zeebe would it be enough to simply change the current environment variables or do we need to also reload the internal service? I guess then it would be without impact anymore.
I found the following in the logs. I have no rights to create container and passed the container for the saving in the connectionstring in the Blobendpoint. He seems to check for a Container and trys to create it.
What is the name of the container he searchs/trys to create.
Or how can i change the containername? As i thought the Basepath variable will be the Containername.
Thanks
Robert
Suppressed: java.lang.Exception: #block terminated with an error
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:104) ~[reactor-core-3.6.8.jar:3.6.8]
at reactor.core.publisher.Mono.block(Mono.java:1779) ~[reactor-core-3.6.8.jar:3.6.8]
at com.azure.storage.common.implementation.StorageImplUtils.blockWithOptionalTimeout(StorageImplUtils.java:146) ~[azure-storage-common-12.26.0.jar:12.26.0]
at com.azure.storage.blob.BlobContainerClient.createIfNotExistsWithResponse(BlobContainerClient.java:374) ~[azure-storage-blob-12.27.0.jar:12.27.0]
at com.azure.storage.blob.BlobContainerClient.createIfNotExists(BlobContainerClient.java:340) ~[azure-storage-blob-12.27.0.jar:12.27.0]
at io.camunda.zeebe.backup.azure.ManifestManager.assureContainerCreated(ManifestManager.java:260) ~[zeebe-backup-store-azure-8.5.6.jar:8.5.6]
at io.camunda.zeebe.backup.azure.ManifestManager.listManifests(ManifestManager.java:213) ~[zeebe-backup-store-azure-8.5.6.jar:8.5.6]
at io.camunda.zeebe.backup.azure.AzureBackupStore.lambda$list$2(AzureBackupStore.java:128) ~[zeebe-backup-store-azure-8.5.6.jar:8.5.6]
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.ThreadPerTaskExecutor$TaskRunner.run(Unknown Source) ~[?:?]
at java.base/java.lang.VirtualThread.run(Unknown Source) ~[?:?]
Can please somebody confirm if delegated SAS Tokens are working and how. For the Accountkey env variable, i get now “‘base64Key’ was not a valid Base64 scheme. Ensure the Storage account key or SAS key is properly formatted”. As the error message says SAS Key would be possible but i have my doubts
The problem seems to be in the Implementation.
In the Manifestmanager.java he ask if the container is created as variable but this variable is initially always false. So he always executes blobContainerClient.createIfNotExists(). This means he will need permissions on storage account level and not only on container level. We will check that.
It can´t be working. Delegated SAS are only for container or storage account. Depending on what you want to get. It cant work that i have only the container rights and on the same time he checks the storage account for the existence of the container what most likely is a storage account permission. Unfortunately this would also be a problem for a normal SAS Account.
So Zeebe is not possible to use SAS Accounts at all. Thats a big security problem for us. Second problem is that delegated SAS expire after some days. How should i change it without redeploying the pod.