You cannot change the REST API - that’s a Camunda implementation. You might be able to change the correlation keys / process variables to match the message…
what you try to achieve is something like transactional message correlation.
One quick and dirty way is the utilisation of the business key (as correlation key) and making it unique in the process instance db table. Problem: Dirty persistance layer exceptions.
Another possibility is the registration of an execution listener that queries for existing process instances with the same business key and throws an exception.
I would like to achieve a unique business key for a particular process. That is, I do not want to have multiple in-flight processes having the same business key, and if there is an attempt to create one, I would like the client to be given an error.
The second options suggested by @jonathan.lukas does get closer to what I was hoping for, but it gives an internal server error (500), still leaving the client without a more specific reason for failure.
I believe I may need the client to query for existing processes with the same business key prior to requesting the creation of a new process, but this will take business logic away from the process and into the client, which is not desirable.
you could extend the Rest API and create an endpoint similar to start process instance.
There, you have the control over checks that can be run before starting a process instance. The main advantage here is that you can control the exact behaviour of the endpoint which is an important requirement. Sorry I underestimated it at first.