S3-Compatible Storage Manifest Write Issue: Chunked Transfer Causing Metadata Corruption

[jamesxql]

When backing up the broker, we selected S3 object storage, but we are using an S3-compatible storage service rather than the native AWS S3. This compatible storage does not support aws-chunked; instead, it directly treats the request body as the object to be stored. Due to this reason, when I call the putObject interface to write the manifest.json metadata, the intended content is:

{JSON content}

But the actual content written becomes:

230
{JSON content}
0
x-amz-checksum-crc32:U6S5Yg==

This causes deserialization errors when retrieving the metadata content via getObject later.

It is recommended to explicitly specify the content-length during putObject and disable chunked transfers:

// S3BackupStore.java
CompletableFuture<ValidBackupManifest> writeManifestObject(final ValidBackupManifest manifest) {
    LOG.debug("Updating manifest of {} to {}", manifest.id(), manifest);
    final AsyncRequestBody body;
    try {
      body = AsyncRequestBody.fromBytes(MAPPER.writeValueAsBytes(manifest));
    } catch (final JsonProcessingException e) {
      return CompletableFuture.failedFuture(e);
    }

     return client
        .putObject(
            request ->
                request
                    .contentLength(body.contentLength().get())
                    .contentType("application/json")
                    .bucket(config.bucketName())
                    .key(objectPrefix(manifest.id()) + MANIFEST_OBJECT_KEY)
                    .build(),
            body)
        .thenApply(resp -> manifest);
  }

[zee-bot]

Hi @jamesxql,

This is a known issue with S3-compatible storage providers that don’t fully support AWS SDK’s chunked transfer encoding and checksum features. The problem you’re experiencing with the corrupted manifest.json files is exactly what’s documented in our troubleshooting guide.

Classification: Problem

You’re encountering manifest corruption due to chunked transfer encoding incompatibility with your S3-compatible storage.

Solution

The recommended fix is to disable the problematic checksum and chunked behavior by setting these environment variables on all your Zeebe brokers:

AWS_REQUEST_CHECKSUM_CALCULATION=WHEN_REQUIRED
AWS_RESPONSE_CHECKSUM_CALCULATION=WHEN_REQUIRED

This disables the automatic checksum calculation and chunked upload behavior that was introduced in AWS SDK 2.30.0, which breaks many S3-compatible stores.

If your S3-compatible provider still requires MD5 checksums, you can also enable legacy MD5 support:

ZEEBE_BROKER_DATA_BACKUP_S3_SUPPORTLEGACYMD5=true

Why This Happens

Some S3-compatible implementations (like Dell EMC ECS) don’t support chunked encoding at all, and others can’t handle the newer checksum features. When chunked encoding is used, the transfer includes metadata like content length and checksums that get written as part of the object content, causing the corruption you’re seeing.

Next Steps

1. Apply the environment variables above to all Zeebe brokers
2. Restart your brokers
3. Re-run your backup operations
4. The manifest files should now be written correctly without the chunked encoding artifacts

Your proposed code solution of explicitly setting contentLength() would also work, but the environment variable approach is the officially supported method that doesn’t require code changes.

References:

• Zeebe S3 Troubleshooting Guide
• S3 Backup Configuration

Let me know if this resolves the issue or if you need any clarification!