We are exploring using lambdas as topic handlers as described in Bernd Ruecker’s Camunda talk: Orchestrate AWS Lambda using Camunda 8. This is appealing from a scalability standpoint, but we have run into issues with invoking larger numbers of lambdas synchronously by the worker. There are limitations imposed by the AWS SDK and after a certain point lambda invocations are queued. Further, it the queue gets long enough you run into SignatureExpired issues when the lambda is finally ready to process.
Has anyone devised a reliable technique for invoking lambdas asynchronously and having each lambda responsible for marking the task complete? We are looking for a method that ensures that all lambdas are successfully invoked and do not run more than once.
If not lambdas, what approaches are used for high-volume enterprise applications?