Http 422 using Telemetry Service

Camunda 7 Topics Discussion & Questions
1

Since a couple of weeks already within my DEV-environment I get the error below every time telemetry data should be send.

An ongoing problem with the service itself or the data sent?
I’m using latest development version of Camunda as well as of Wildfly.

29.12. 07:18:15,770 DEBUG [org.camunda.bpm.engine.telemetry#logDebug] ENGINE-28006 Sending telemetry data: 
29.12. 07:18:15,805 WARN  [org.camunda.bpm.engine.telemetry#logWarn] ENGINE-28002 Could not send telemetry data. Reason: ProcessEngineException with message 'ENGINE-28004 Unexpected response code 422 when sending telemetry data'. Set this logger to DEBUG/FINE for the full stacktrace.
29.12. 07:18:15,807 DEBUG [org.camunda.bpm.engine.telemetry#logDebug] ENGINE-28003 org.camunda.bpm.engine.ProcessEngineException occurred while sending telemetry data.: org.camunda.bpm.engine.ProcessEngineException: ENGINE-28004 Unexpected response code 422 when sending telemetry data ...
	at org.camunda.bpm.engine.impl.telemetry.TelemetryLogger.unexpectedResponseWhileSendingTelemetryData(TelemetryLogger.java:43)
	at org.camunda.bpm.engine.impl.telemetry.reporter.TelemetrySendingTask.sendData(TelemetrySendingTask.java:252)
	at org.camunda.bpm.engine.impl.telemetry.reporter.TelemetrySendingTask.run(TelemetrySendingTask.java:138)
	at java.util.TimerThread.mainLoop(Timer.java:555)
	at java.util.TimerThread.run(Timer.java:505)

Data sent after startup this morning:

{
    "installation": "483d1d4e-15df-11eb-adc1-0242ac120002",
    "product": {
        "name": "Camunda BPM Runtime",
        "version": "7.15.0-SNAPSHOT",
        "edition": "community",
        "internals": {
            "database": {
                "vendor": "Informix Dynamic Server",
                "version": "12.10.FC12WE"
            },
            "application-server": {
                "vendor": "WildFly",
                "version": "WildFly EE 22.0.0.Final-SNAPSHOT (WildFly Core 14.0.0.Beta5-SNAPSHOT) - 2.2.3.Final"
            },
            "license-key": null,
            "commands": {
                "TelemetrySendingTask$$Lambda$1339/56074987": {
                    "count": 3
                }
            },
            "camunda-integration": [
                "wildfly-subsystem"
            ],
            "metrics": {
                "root-process-instance-start": {
                    "count": 0
                },
                "unique-task-workers": {
                    "count": 0
                },
                "activity-instance-start": {
                    "count": 0
                },
                "executed-decision-elements": {
                    "count": 0
                },
                "executed-decision-instances": {
                    "count": 0
                }
            },
            "webapps": [],
            "jdk": {
                "version": "1.8.0_272",
                "vendor": "AdoptOpenJDK"
            },
            "telemetry-enabled": true
        }
    }
}
2

Hi Frank,

Thanks for reporting this, I’ll try to figure out what the cause of the problem is. Did you modify any of the telemetry-related engine configuration options?

Cheers,
Thorben

1 Like
3

Hi Thorben,

I just enabled Telemetry by setting

    <property name="initializeTelemetry">true</property>

in standalone-full.xml and enabled debug-logging:

    <logger category="org.camunda.bpm.engine.telemetry">
        <level name="DEBUG"/>
    </logger>

Nothing else I’m aware of.

Regards, Frank

1 Like
4

I can’t reproduce it with the current Wildfly distribution snapshot from my location. In my case, the following JSON is submitted successfully:

{
  "installation": "b856be9f-f39a-4d29-909b-e4e85f7b60d4",
  "product": {
    "name": "Camunda BPM Runtime",
    "version": "7.15.0-SNAPSHOT",
    "edition": "community",
    "internals": {
      "database": {
        "vendor": "H2",
        "version": "1.4.190 (2015-10-11)"
      },
      "application-server": {
        "vendor": "WildFly",
        "version": "WildFly Full 20.0.1.Final (WildFly Core 12.0.3.Final) - 2.1.3.Final"
      },
      "license-key": null,
      "commands": {
        "DbMetricsReporter$1": {
          "count": 12
        },
        "AcquireJobsCmd": {
          "count": 24
        },
        "TelemetrySendingTask$$Lambda$1371/494020140": {
          "count": 1
        },
        "TelemetryConfigureCmd": {
          "count": 1
        },
        "TelemetrySendingTask$$Lambda$1349/1680257707": {
          "count": 1
        },
        "IsTelemetryEnabledCmd": {
          "count": 1
        },
        "HistoryCleanupHandler$1": {
          "count": 4
        },
        "ExecuteJobsCmd": {
          "count": 4
        },
        "SuccessfulJobListener": {
          "count": 4
        }
      },
      "camunda-integration": [
        "wildfly-subsystem"
      ],
      "metrics": {
        "root-process-instance-start": {
          "count": 0
        },
        "unique-task-workers": {
          "count": 1
        },
        "activity-instance-start": {
          "count": 0
        },
        "executed-decision-elements": {
          "count": 0
        },
        "executed-decision-instances": {
          "count": 0
        }
      },
      "webapps": [
        "welcome"
      ],
      "jdk": {
        "version": "1.8.0_242",
        "vendor": ""
      },
      "telemetry-enabled": true
    }
  }
}

I’ll check with our telemetry team next week if we can tell from server-side what is going on.

5

Hello Frank,

The 422 response is return due to invalid version of the installation UUID, all installation IDs should be version 4 UUID.
The installation Id is created during the initial start of the process engine and stored in the ACT_GE_PROPERTY table of the Camunda database.
So I would like to ask you:

  • Did you adjust manually the installation ID?
  • Which is the jdk that you are using? In the implementation we are using the UUID.randomUUID() where I don’t expect differences in the behavior as by javadoc the type is 4.

Best regards,
Yana

6

Hi Yana,

I replaced my Installation UUID with a version 4 UUID and now sending of telemetry data works again.

The creation of the UUID during first startup does not work for me when using the IBM Informix database.
A problem described in https://jira.camunda.com/browse/CAM-8282 some time ago.

So I inserted this record into table act_ge_property manually and do not remember from where I took this UUID and was not aware, that there are different types of UUIDs and that they are checked by Camunda.

7

Hi Frank,

I see, thank you for the clarification, we can consider improving the documentation to specify that installation id should be UUID type 4 that might prevent the issue you ran into.
I remember CAM-8282, unfortunately we use such locks as mechanism to prevent concurrent issues on cluster scenarios.

Best regards,
Yana

8