Pitfalls with JSON data in process variables

Camunda 7 Topics Discussion & Questions
1

In a sample project we tried to use an external service task handled by the JavaScript task client, using v7.10 and camunda-external-task-client-js v1.1.1. The client was supposed to subscribe a topic and poll Camunda.
Our task client tried to pass a process variable containing a json string of roundabout 17KB - a list of news items.

We were unable to let the js taskclient send JSON data like below:

const {Client, logger, Variables} = require("camunda-external-task-client-js"),
request = require('request'),
path = require('path'),
camCommon = require(path.resolve(__dirname, "./common.js"));
//  rest url is for spring-boot, differs from war
const config = {baseUrl: "http://localhost:8095/rest", 
   use: logger, asyncResponseTimeout: 5000};
const client = new Client(config);

client.subscribe("news", async function ({task, taskService}) {
    const options = {
        url: 'https://example.com/api/news',
        method: 'GET'
    };

    request(options, function (err, res, body) {
        const processVariables = new Variables();
        processVariables.set("news", JSON.parse(body);
        taskService.complete(task, processVariables);
    });
});

due to this error:

019-01-27 11:52:49.722  WARN 22306 --- [nio-8095-exec-9] ExceptionHandler                         : org.camunda.bpm.engine.rest.exception.RestException: Unsupported value type 'json'
at org.camunda.bpm.engine.rest.dto.VariableValueDto.toTypedValue(VariableValueDto.java:94)
at org.camunda.bpm.engine.rest.dto.VariableValueDto.toMap(VariableValueDto.java:153)

The js taskclient readme points to About JSON & Date Variables which leads you to believe that the client can pass JSON. However it can’t, because JSON is not a known type in ValueTypeResolverImpl:

public ValueTypeResolverImpl() {
  addType(BOOLEAN);
  addType(BYTES);
  addType(DATE);
  addType(DOUBLE);
  addType(INTEGER);
  addType(LONG);
  addType(NULL);
  addType(SHORT);
  addType(STRING);
  addType(OBJECT);
  addType(NUMBER);
  addType(FILE);
}

Is that an error in the documentation or did we miss something? I know that the browser can define variables of type json and submit them to the engine, shouldn’t external task clients or clients posting data to post-message be able to do likewise?

In an attempt to work around this, we changed our code so that the variable was not a JavaScript object, but a plain json String.

That failed, too, apparently because Camunda tries to persist the entire JSON in a database table which doesn’t have enough space for the data.

Caused by: org.h2.jdbc.JdbcBatchUpdateException: Value too long for column "TEXT_ VARCHAR(4000)"

So the limit for string variables is VARCHAR(4000).

I have seen Can't set a large size file as a process variable!, but a 17kB JSON does not seem to be an excessive amount of data for backend operations.

One possible workaround is to define a Java bean which matches the JSON and pass a variable of type “object” with a value that is a JSON String from the taskclient (also see related issue)

But Is there any advice for pure JSON lovers?

2 Likes
2

Shouldn’t you do this per the example?

Instead of this:

processVariables.set("news", JSON.parse(body);

Do this?

processVariables.set("news", body);

(Just looking at the XML example, it looks like they forgot the variable name param in the JSON example).

processVariables.setTyped("news",  {
  type: "json",
  value: body,
  valueInfo: {}
});

We’ve struggled and struggled with XML for about 2-3 years now, and I just recently figured out how to get it to store as an XmlValue and JsonValue (to get past the varchar2(4000) for strings.

3

Hi @dschulten,

please consider the following simple example:

const {
  Client,
  logger,
  Variables
} = require("camunda-external-task-client-js");

// bootstrap the client
const client = new Client({
  baseUrl: "http://localhost:8080/engine-rest",
  use: logger
});

// subscribe to the topic 'aTopicName'
client.subscribe("aTopicName", async ({ task, taskService }) => {
  // my JSON Object
  const jsonObject = [9, 1, 4, 10, { aKey: "aValue" } ];

  // set process variable 'jsonObject'
  const processVariables = new Variables()
    .set("jsonObject", jsonObject);

  // complete the task
  try {
    await taskService.complete(task, processVariables);
    console.log("I completed my task successfully!!");
  } catch (e) {
    console.error(`Failed completing my task, ${e}`);
  }
});

Whenever you set a variable value without a type information, the External Task Client performs a type lookup. In the example shown above, a JSON value is set which will eventually be stored as variable of type ‘Json’.

Does this answer your question?

Cheers,
Tassilo

4