Script Based unit testing with Coverage Reporting (Spock Framework)

[StephenOTT]

Hey all

Wanted to share a work in progress:

This is a implementation of Spock Framework as discussed on:

GitHub - DigitalState/Camunda-Spock-Testing: Spock Framework Specification...

But we are now taking it further to add BPMN Coverage testing similar to:

GitHub - camunda-community-hub/camunda-bpm-process-test-coverage: Community...

But the coverage library i have found to generally be complicated to extend and was very confusing on how it worked. The goal here was to open coverage testing to “everyone” / even the less technical bpmn devs.

So we end up with

some differences between the current BPMN coverage lib:

1. It works with the default engine; no custom bean configs required
2. it uses all of the built in camunda engine services to generate the required data; there is no custom data models being stored.
3. it uses all of the in-house spock framework features and the spock-reports extension, providing really easy extendability.

You can follow some of the dev discussions here:

Rebuild alternative using Spock Framework

Or feel free to ask questions or get involved.

[StephenOTT]

Update with more visual support:

BPMN xml, activity instances, sequence flow instances, User Tasks, and Async Configs for all elements.

[StephenOTT]

Okay updated with Multi-Feature support

So this is one Spec with multiple Features

[StephenOTT]

Updated with Mult-Feature and Multi-Iterations Per Feature using Spock Data Tables.

Notice how there are multiple Coverage Maps in the second Feature. This is a Coverage Map for Each Iteration

@nikku If you have any suggestions on best ways to optimize loading of many BPMN viewers on a single page that would be appreciated.

click on image to see full length

Notice how Iteration Coverage have Iteration Counts that align with the Iteration Counts in the Data Table.

[StephenOTT]

Here is a update image of Multi-Iterations with different coverage outcomes per iteration:

click image to see full length, and scroll to bottom of image

[StephenOTT]

New update for further refinements and better theme

click on image to see full length

[StephenOTT]

another usage of bootswatch

[StephenOTT]

Some more UI cleanups for easier use:

1. Uses BPMN Navigated Viewer allowing scrolling and zoom on each of the BPMN diagrams (Thanks @nikku)

2. Supports Spocks @Unroll feature allowing Data tables to be run as independent feature runs.

3. More clear Pass Fail

4. Bootstrap cleanup

5. Narrative cleanup

6. Fully Responsive support so it can be easily shared.

7. All dependencies are loaded through CDN (BPMN-JS, bootswatch, logos, etc) The goal is that it is 1 file you can easily share with whomever and it will open on their browser.

8. Support for duplicate Data table usage (every BPMN diagram is a unique ID)

Enjoy

click on image to view full length

@thorben thanks for the visuals feedback

[StephenOTT]

Okay some good updates today on this:

Coverage has not been added as a dynamic action that can be added that any point in a test:

So at anypoint you can do:

reportInfo(
   generateCoverageData(
                        processEngine(), 
                        callActivityCoverage1ProcessInstance, 
                        "CallActivityCoverage1.bpmn Completion"
                        )
)

The three inputs are the processEngine instance to run the query against (has interesting implications for Multi-Engine testing), the specific ProcessInstance to get the coverage for, and the human name for the coverage diagram.

You Can run this at any time and multiple times / as you see fit for your purposes.

You get output such as:

Single Feature usage:

Multi-feature usage:

@felix-mueller

[StephenOTT]

I have updated the coverage library to be a stand alone coverage generator that lets you generate coverage using Groovy traits and a few methods.

GitHub - DigitalState/camunda-coverage-generation: Camunda BPM Unit testing...

FYI as might be of interest for easy to generate coverage throughout a process lifecycle @Niall @felix-mueller.

Here is a spock example of it in action:

class CallActivitySingleFeatureSpec extends Specification implements CoverageBuilder, SequenceFlowHistory {

    @ClassRule
    @Shared ProcessEngineRule processEngineRule = new ProcessEngineRule('camunda_config/camunda.cfg.xml')
    @Shared String deploymentId

    def setupSpec(){
        def deployment = repositoryService().createDeployment()
                .addInputStream(getSequenceFlowFileName(), getSequenceFlowListenerScript())
                .addModelInstance('CallActivityCoverage.bpmn', addSequenceFlowListeners('bpmn/CallActivityCoverage.bpmn'))
                .addModelInstance('CallActivityCoverage2.bpmn', addSequenceFlowListeners('bpmn/CallActivityCoverage2.bpmn'))
                .name('CallActivitiesCoverage')
                .enableDuplicateFiltering(false)
                .deploy()
        deploymentId = deployment.getId()
        println "Deployment ID: '${deploymentId}' has been created"
    }

    def 'Manage CallActivityCoverage1'() {
        when: 'Setting up variables'
        def json = S("{\"customer\": \"Kermit\"}")
        def startingVariables = [
                'json': json
        ]

        and: 'We start the CallActivityCoverage process definition'
        ProcessInstance callActivityCoverage1ProcessInstance = runtimeService().startProcessInstanceByKey('CallActivityCoverage')

        then: 'Process is Active and waiting for user task completion'
        assertThat(callActivityCoverage1ProcessInstance).isActive()

        then: 'The current process variables are equal to the starting variables'
        def processVariables = runtimeService().getVariables(callActivityCoverage1ProcessInstance.getProcessInstanceId())
        assertThat(processVariables == startingVariables)

        then: 'The process instance should be waiting for the Call Activity to Complete'
        assertThat(callActivityCoverage1ProcessInstance).isWaitingAt('Task_1gdn63n')

        coverageSnapshot(callActivityCoverage1ProcessInstance, 'some1')

        and: 'get the called called process instance'
        HistoricActivityInstance callActInstance = historyService().createHistoricActivityInstanceQuery()
                .processInstanceId(callActivityCoverage1ProcessInstance.getProcessInstanceId())
                .activityId('Task_1gdn63n')
                .singleResult()

        ProcessInstance callActivityCoverage2ProcessInstance = calledProcessInstance(processInstanceQuery().processInstanceId(callActInstance.getCalledProcessInstanceId()))

        then: 'CallActivityCoverage2 is running'
        assertThat(callActivityCoverage2ProcessInstance).isActive()

        then: 'CallActivityCoverage2 is waiting at the User Task'
        assertThat(callActivityCoverage2ProcessInstance).isWaitingAt('Task_0xjkfyv')

        then: 'Complete the User Task'
        complete(task(callActivityCoverage2ProcessInstance))

        then: 'CallActivityCoverage2 has completed'
        assertThat(callActivityCoverage2ProcessInstance).isEnded()

        coverageSnapshot(callActivityCoverage2ProcessInstance, 'some2')

        then: 'CallActivityCoverage1 has ended'
        assertThat(callActivityCoverage1ProcessInstance).isEnded()

        coverageSnapshot(callActivityCoverage1ProcessInstance)
    }

    def cleanupSpec() {
        saveCoverageSnapshots()

        repositoryService().deleteDeployment(deploymentId,
                true, // cascade
                true, // skipCustomListeners
                true) // skipIoMappings
        println "Deployment ID: '${deploymentId}' has been deleted"
    }

}

notice the coverageSnapshot() and the saveCoverageSnapshots() methods.

[StephenOTT]

As per discussion on https://github.com/DigitalState/camunda-coverage-generation-groovy/issues/9, I have added a new class to the coverageGenerator which now allows JUnit and pure Java usage. Groovy is still a hard dep for under the hood operations, but you can now use the Coverage Generator directly from your Java Unit Tests

See: https://github.com/DigitalState/camunda-coverage-generation-groovy#using-the-coverage-builder-with-junit--pure-java

package coveragetest;

import io.digitalstate.camunda.coverage.bpmn.CoverageBuilderJavaBridge;
import org.camunda.bpm.engine.runtime.ProcessInstance;
import org.camunda.bpm.engine.test.Deployment;
import org.camunda.bpm.engine.test.ProcessEngineRule;

import static org.camunda.bpm.engine.test.assertions.ProcessEngineTests.*;

import org.junit.Rule;
import org.junit.Test;

/**
 * @author Daniel Meyer
 * @author Martin Schimak
 */
public class SimpleTestCase {

    @Rule
    public ProcessEngineRule rule = new ProcessEngineRule("camunda_config/camunda.cfg.xml");
    CoverageBuilderJavaBridge coverageBuilder = new CoverageBuilderJavaBridge();

    @Test
    @Deployment(resources = {"testProcess.bpmn"})
    public void shouldExecuteProcess() {
        // Given we create a new process instance
        ProcessInstance processInstance = runtimeService().startProcessInstanceByKey("testProcess");
        // Then it should be active
        assertThat(processInstance).isActive();
        // And it should be the only instance
        assertThat(processInstanceQuery().count()).isEqualTo(1);
        // And there should exist just a single task within that process instance
        assertThat(task(processInstance)).isNotNull();

        // When we complete that task
        complete(task(processInstance));
        // Then the process instance should be ended
        assertThat(processInstance).isEnded();

        coverageBuilder.coverageSnapshot(processInstance);
        coverageBuilder.saveCoverageSnapshots();

    }

}

and remember you can do interesting coverage generation scenarios such as:

package coveragetest;

import io.digitalstate.camunda.coverage.bpmn.CoverageBuilderJavaBridge;
import org.camunda.bpm.engine.runtime.ProcessInstance;
import org.camunda.bpm.engine.test.Deployment;
import org.camunda.bpm.engine.test.ProcessEngineRule;

import static org.camunda.bpm.engine.test.assertions.ProcessEngineTests.*;

import org.junit.Rule;
import org.junit.Test;

/**
 * @author Daniel Meyer
 * @author Martin Schimak
 */
public class SimpleTestCase {

    @Rule
    public ProcessEngineRule rule = new ProcessEngineRule("camunda_config/camunda.cfg.xml");
    CoverageBuilderJavaBridge coverageBuilder = new CoverageBuilderJavaBridge();

    @Test
    @Deployment(resources = {"testProcess.bpmn"})
    public void shouldExecuteProcess() {
        // Given we create a new process instance
        ProcessInstance processInstance = runtimeService().startProcessInstanceByKey("testProcess");
        // Then it should be active
        assertThat(processInstance).isActive();
        // And it should be the only instance
        assertThat(processInstanceQuery().count()).isEqualTo(1);
        // And there should exist just a single task within that process instance
        assertThat(task(processInstance)).isNotNull();

        coverageBuilder.coverageSnapshot(processInstance);

        // When we complete that task
        complete(task(processInstance));
        // Then the process instance should be ended
        assertThat(processInstance).isEnded();

        coverageBuilder.coverageSnapshot(processInstance);
        coverageBuilder.saveCoverageSnapshots();

    }

}

which would generate two snapshots: first would be before the User Task is completed, and the second would be after the process instance has completed.