Freemarker Script: Greater than 4000 char, exceeding historic variable instance DB insert

StephenOTT · May 11, 2017, 2:25pm

I am working with a freemarker script that is greater than 4000 char, and it is currently breaking the DB insert for historic variables.

The freemarker script is currently being run as a Input parameter. Can the javascript engine call the freemarker engine? I was thinking of storing the output of the freemarker script as SPIN json, but cant see a way to call the freemarker engine from nashorn.

@camunda
@thorben thoughts?

Thanks

thorben · May 11, 2017, 2:32pm

Hi Stephen,

Since you can call any Java code from Nashorn, you can use the JSR-223 API to obtain the scripting engine and then execute a script. I think the freemarker engine’s name is freemarker.

Cheers,
Thorben

StephenOTT · May 11, 2017, 3:04pm

Okay, got the engine to execute. But the Camunda execution context is lost, Ideas on ability to pass the camunda execution context into the engine so that the freemarker eval can still render things like ${myBPMNProcessVariable}

here is a code snippet so far:

var ScriptEngine= new JavaImporter(javax.script);

with (ScriptEngine) {

   var manager = new ScriptEngineManager();
   var engine = manager.getEngineByName("freemarker");

   var rawValue = 'Dear ${myProcessVariable},Welcome to camunda BPM. Please visit one of these links for more information about camunda BPM';

   var rendered = engine.eval(rawValue);

   var json = {
      "content": rendered
      }

   S(json);

currently looking at: https://docs.oracle.com/javase/7/docs/api/javax/script/ScriptEngine.html for additional options for the eval()

thorben · May 11, 2017, 3:23pm

You will need an instance of javax.script.Bindings and pass that to #eval to resolve context variables. If you want to dynamically resolve all execution variables, using internal API is the least painful way to do this properly, e.g.

ScriptEngine engine = ...;
DelegateExecution execution = ...;

ScriptingEngines scriptingEngines = Context.getProcessEngineConfiguration().getScriptingEngines();
Bindings bindings = scriptingEngines.createBindings(engine, execution);

.. eval

The above is Java syntax but should be easy to adapt to Nashorn-style Javascript.

StephenOTT · May 11, 2017, 3:42pm

@thorben can you explain what is going on in:

ScriptingEngines scriptingEngines = Context.getProcessEngineConfiguration().getScriptingEngines();
Bindings bindings = scriptingEngines.createBindings(engine, execution);

based on my understanding of your snippet, i should be able to do engine.createBindings() ??

My first attempt was:

   var camundaExecution = execution;

   var manager = new ScriptEngineManager();
   var engine = manager.getEngineByName("freemarker");

   engine.createBindings(engine,camundaExecution);

but the createBindings fails as createBinding does not seem to function the same based on the docs: https://docs.oracle.com/javase/7/docs/api/javax/script/ScriptEngine.html#createBindings()

StephenOTT · May 11, 2017, 5:32pm

@thorben as a temp work around i have been doing the following:

var processDefinitionId = execution.getProcessDefinitionId();

var deploymentId = execution.getProcessEngineServices().getRepositoryService().getProcessDefinition(processDefinitionId).getDeploymentId();

var resource = execution.getProcessEngineServices().getRepositoryService().getResourceAsStream(deploymentId, 'content.txt');

var IOUtils = Java.type('org.apache.commons.io.IOUtils');
var String = Java.type('java.lang.String');

var contentFile= new String(IOUtils.toByteArray(resource), 'UTF-8');


   var placeholderValues= {
      "key1": execution.getVariable('val1').prop("val1").value(),
      "key2": execution.getVariable('val2').prop("val2").value(),
      "key3": execution.getVariable('val3').prop("val3").value()
   }


var ScriptEngine= new JavaImporter(javax.script);

with (ScriptEngine) {

   var manager = new ScriptEngineManager();
   var engine = manager.getEngineByName("freemarker");

   var bindings = engine.createBindings();

   bindings.put("placeholders", placeholderValues);

   var rendered = engine.eval(contentFile, bindings);

   var json = {
      "content": rendered
   }

   S(JSON.stringify(json));

};

BUT i have seeing a weird issue i cannot seem to identify where var IOUtils = Java.type('org.apache.commons.io.IOUtils'); is not able to load. i get: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.commons.io.IOUtils

thoughts?

StephenOTT · May 11, 2017, 10:09pm

Okay. So not sure what was causing the IOUtils to not function…

but here is a working code example for anyone going forward:

So in this use case i was dealing with a large email that needed to be rendered by freemarker

The html content is stored in a file called content.txt. You can see this on line 3 of the script.

var processDefinitionId = execution.getProcessDefinitionId();
var deploymentId = execution.getProcessEngineServices().getRepositoryService().getProcessDefinition(processDefinitionId).getDeploymentId();
var resource = execution.getProcessEngineServices().getRepositoryService().getResourceAsStream(deploymentId, 'content.txt');

var IOUtils = Java.type('org.apache.commons.io.IOUtils');
var String = Java.type('java.lang.String');

var content = new String(IOUtils.toByteArray(resource), 'UTF-8');

// Placeholder content that will be binded into the freemarker bindings.
// in Freemarker, you access these with ${placeholders.firstName}
var placeholderValues= {
   "firstName": "Someones First Name",
   "id": "123456789",
   "time": "5",
   "name": "A Name"
}

// Render the html into the freemarker engine
var ScriptEngine= new JavaImporter(javax.script);

var emailContent = "";

with (ScriptEngine) {
   var manager = new ScriptEngineManager();
   var engine = manager.getEngineByName("freemarker");

   var bindings = engine.createBindings();
   bindings.put("placeholders", placeholderValues);

   var rendered = engine.eval(content, bindings);
   emailContent = rendered;

};

The emailContent variable will contain the rendered/processed content

StephenOTT · May 12, 2017, 7:39pm

Cross-Referenced Bug report: org.apache.commons.io.IOUtils cannot be found when task runs as Job

StephenOTT · May 17, 2017, 2:12pm

Solved in org.apache.commons.io.IOUtils cannot be found when task runs as Job. (see that post for more info about why the previous code in this thread fails)

Working code snippet allowing you to load a resource from a deployment:

var processDefinitionId = execution.getProcessDefinitionId();
var deploymentId = execution.getProcessEngineServices().getRepositoryService().getProcessDefinition(processDefinitionId).getDeploymentId();
var resource = execution.getProcessEngineServices().getRepositoryService().getResourceAsStream(deploymentId, 'content.txt');

var Scanner= Java.type("java.util.Scanner");
scannerResource = new Scanner(resource, "UTF-8");

var content = scannerResource.useDelimiter("\\Z").next();

scannerResource.close();

Where content is the variable with your resource’s content

mhelling · January 15, 2019, 12:46pm

Hi,

sorry to revive this old thread, but I stumbled across this exact same issue now. I understand both the problem
and the “solution” - which is in my eyes merely a hacky workaround I cannot expect customers or integrators to implement. I am using a template engine in order to enable non-programmers a certain flexibility. I also cannot use resources in deployments, which would mean I had to escape the template before assigning it to a variable only to then pass it to the template engine.

As I see it there are two senseful ways of “dealing” with it:

Just truncate the string and insert it nonetheless (would be OK for me since I do not rely on historic data). Probably this should be opt-in to prevent unexpected results. Just add an xml attribute like “ignoreTruncation”.
Put the data in ACT_GE_BYTEARRAY just like you do it with non-historic data

This is not a priority issue for me, but I would like to see this at least in the backlog.

Thanks and greetings,
Matt

EDIT: Okay, after I installed a workaround for the truncation by ignoring it on database level it seems that this not only affects the history tables, but also the execution tables. This of course changes the whole matter quite a lot since I of course need a non-truncated value in the execution. When using the REST api I can tell it to interpret the string as JSON, thus object, which will store it in the byte array table. Cannot do this with variables defined in the BPM file afaik. But I need it right there…

StephenOTT · January 15, 2019, 6:10pm

@mhelling why is this a hacky workaround?
Where are you storing your 4000+ character templates?

mhelling · January 15, 2019, 8:14pm

I’m not storing them directly, but I create dynamic JSON documents using a template that may result in more than 4000 chars (depending on end-user input). I’m then using the document to feed a REST endpoint.

StephenOTT · January 15, 2019, 8:17pm

So you can build your template and json in memory, thus no 4000 limit; and then you can store your template as escaped JSON / the json your will send into your REST endpoint. Use the Camunda SPIN library to store your template in a json object and thus you will not have the 4000k string limit.

mhelling · January 15, 2019, 8:25pm

Not without the drawbacks as outlined in my first post. I want to use the template engine to simplify things, not to make them more difficult and less maintainable. Yes, I as a programmer could do it, but my (trained) customers and business integrators should be able to use an existing toolchain including templates. Not sure if I’m expressing my concers clearly?

Btw. I am using Camunda as a blackbox only via REST.

StephenOTT · January 15, 2019, 8:28pm

You can use the template engine: see: Freemarker Script: Greater than 4000 char, exceeding historic variable instance DB insert you can use the

 var manager = new ScriptEngineManager();
   var engine = manager.getEngineByName("freemarker");

   var bindings = engine.createBindings();
   bindings.put("placeholders", placeholderValues);

   var rendered = engine.eval(content, bindings);
   emailContent = rendered;

The above is doing the same thing as typing “Freemarker” in the script type field of the script task, but you are doing it all in memory and thus not saving anything to the DB as a String.

When you run var rendered = engine.eval(content, bindings);, you just save the rendered as a JSON Spin variable, and it will save as a byte array in the DB and thus not have the 4k limit. All of this is invisible to any end client, and invisible to the Rest API.

You can also write this as a small reusable method that you can ship with your camunda instance, and import in your script

mhelling · January 15, 2019, 8:35pm

One step back please. From my point of view I want to do the following:

Open up Camunda Modeler
Load my BPMN-File or create a new one
Create a service task
Add a variable with script type freemarker
Paste my freemarker template in the script textarea
Save the file.

I shouldn’t be required to even care if the resulting string is less or more than 4k chars long. As I said, I understand both the problem AND the solution you posted. I would have to escape my freemarker template, put it in a variable, use the script above, store the result string in yet another variable. Same output, but infeasible if you want to give non-programmers the tools (which exist and work pretty well in all other aspects) to make or modify their own processes.

Btw. thanks for your help and patience. Don’t get me wrong

StephenOTT · January 15, 2019, 9:55pm

Then i would use a Camunda Delegate to create a pre-built delegate that accepts a String, and converts that into a Spin JSON object.

The Script Task saves its result as a String. This is not a freemarker issue, this is just the desired and expected result of Script Tasks and using the Result Variable feature of script tasks.

If your non-developers are using the Camunda Modeler, then create a Modeler Element Template that implements the Java delegate described above. You can then your non-developers just need to pick from the template and fill out the Template’s inputs.

mhelling · January 16, 2019, 8:10am

And how would I do that without java programming? I use Camunda as a blackbox in Docker and only via the REST api. My software is written in .NET. If it were possible only using the BPMN file - okay - accepted as workaround. But still - hacky.

The only way is to properly implement this in Camunda itself. Just give me the option to tell the engine that the string is to be interpreted as JSON or LONGTEXT (for your e-mail use case) and we’re totally fine. No hacks required.

JanTichava · January 30, 2019, 12:34pm

Hello Stephen, I use Camunda Modeler template to fill InputParameter (very long HTML message). The InputParameter is defined as freemarker inline script. Same as you I have String variable length limitation problem (since freemarker automatically renders output to Camunda String variable type). So I am thinking about manual freemarker rendering, but to a File.

So to begin I wrote custom class with static method, which transforms input String to File and returns it:

package com.test;

import org.camunda.bpm.engine.variable.Variables;
import org.camunda.bpm.engine.variable.value.FileValue;
import org.camunda.bpm.engine.variable.value.builder.FileValueBuilder;

import java.io.UnsupportedEncodingException;

public class MyUtils {

public static FileValue ConvertToFile(String inputString) throws UnsupportedEncodingException {
    FileValueBuilder fileValueBuilder = Variables.fileValue("test.txt");

    FileValue fileValue = fileValueBuilder.file(inputString.getBytes("UTF-8"))
            .mimeType("text/plain")
            .encoding("UTF-8")
            .create();

    return fileValue;
}

}

Then the InputParameter will be defined as groovy inline script like:

com.test.MyUtils.ConvertToFile("My sample ${template} to be processed inside method above later.")

It works fine, but problem is that user has to place everything to the method (and escape some chars). Which he has to know obviously.

My template is defined like:

{
     "label": "Message",
	 "description": "(FreeMarker script is allowed, e.g. ${MyVariable}).",
	 "type": "Text",
	 "value": "",
	 "binding": {
	     "type": "camunda:inputParameter",
         "name": "HTMLMessage",
         "scriptFormat": "groovy"
	  },
	  "constraints": {
	     "notEmpty": true
	  }
}

If we could specify something like below, problem will be solved for me:

{
     "label": "Message",
	 "description": "(FreeMarker script is allowed, e.g. ${MyVariable}).",
	 "type": "Text",
	 "value": "com.test.MyUtils.ConvertToFile(" + value + ")",
	 "binding": {
	     "type": "camunda:inputParameter",
         "name": "HTMLMessage",
         "scriptFormat": "groovy"
	  },
	  "constraints": {
	     "notEmpty": true
	  }
}

Do you have any idea how to solve it?

Thanks, Jan.

StephenOTT · January 31, 2019, 5:10pm

I would recommend that you make your Method look for local variables.

So within your method’s execution you have stuff like execution.getVariableLocal(..) and these are like “pre-set” variable names that are expected.

So your template content would be a local input variable that is picked up by your method.
Your method would not really have any arguments. (i guess you could make one of the arguments the ‘local variable name’ you want the method to use.)