Compensate ProcessInstance after Incident appeared

My Team and I need some help for the best practice regarding how to compensate a ProcessInstance in Camunda after an Incident happened.

Situation:
Cause the IncidentHandler triggers before the Incident is completely created and stored in the ProcessEngine we choose to have a SchedularJob which executes every 2 seconds insted of using an IncidentHandler.

  1. Within its execution it searches for all incidents and their ProcessInstances.

  2. For every found Incident with an active ProcessInstance, the SchedularJob sends a Message/Signal to the corresponding ProcessInstance.

  3. The BPM of the ProcessInstance contains an EventSubprocess with a SignalStartEvent followed by a CompensationIntermediateThrowEvent to trigger all compensations of the ProcessInstance.

Problems:

  1. After an execution of the SchedulerJob the Incident is not resolved/removed, so that in the next execution of the SchedulerJob the same Incident is found again and its ProcessInstance will start its compensation again.
  2. Incidents of the type “FailedJob” are not resolvable.
  3. New Incidents can appear within the triggered compensation of the ProcessInstance, which ends up in a loop of compensations triggered by the SchedulerJob.

Questions:

  1. What is the best practice to manage Incidents, which are already handled by running the compensation of the ProcessInstance.

  2. How to avoid endless loops caused by not resolved Incidents or Incidents which happend within the compensation?

  3. How should we handle Incidents of the type “FailedJob”? Are customized Incidents a thing? Maybe we should change failedJob-Incidents in customized Incidents differed by “Incident in Compensation” and “Incident in regular Process”?

Many thanks for every help.