Replacing Http-Connector with Jsoup usage

Hey everyone

so use case came up recently for doing some web scraping with Camunda. From this we needed more powerful features from HTTP requests and so we loaded into camunda int eh shared engine.

From this we realized that this was actually a much more flexible solution compared to using HTTP-connector.

Jsoup is optimized for html/xml responses, but with a few tweaks to the configurations of a request (all standard/supported by Jsoup), we end up with a great JSON or “whatever” http request manager for requests and responses.

It also has many fixes for other problems with http-connect such as timeouts, attachments/binary data, buffer streams, and response time, etc. (many of which have been discussed in detail throughout the forum).

So what did we end up with!:

First we added Jsoup to Camunda. We put together a docker compose for you to make this as easy as possible:

Specifically: camunda-variations/web-scrape at master · DigitalState/camunda-variations · GitHub

This will load camunda 7.7 tomcat with deployment-aware = false and add Jsoup to the classpath


In our case we use Javascript/nashorn for our scripting

So you can easily access Jsoup through Java or just through the script engine:


with (new JavaImporter(org.jsoup))

  // var body = {
  //   "myKey1": "myValue1",
  //   "myKey2": "myValue2",
  //   "myKey3": {
  //     "internal1":"internalV1",
  //     "internal2":"internalV2"
  //     },
  //   "myKey4": [
  //     1,2,3,4,5
  //     ]
  // }

  var doc = Jsoup.connect('')
                  // .method(Java.type('org.jsoup.Connection.Method').POST)
                  .header('Accept', 'application/json')
                  .header('Content-Type', 'application/json')
                  // .data('filterABC', 'subgroup1')
                  // .requestBody(JSON.stringify(body))
                  .ignoreContentType(true) // This is used because Jsoup "approved" content-types parsing is enabled by default by Jsoup

  var resBody = doc.body()
  var resStatusCode = doc.statusCode()
  var resStatusMessage = doc.statusMessage()
  var resContentType = doc.contentType()
  var resCharSet = doc.charset()


function spinify(body)
  var parsed = JSON.parse(body)
  var stringified = JSON.stringify(parsed)
  var spin = S(stringified)
  return spin

execution.setVariable('responseBodyString', spinify(resBody))

If you take a look through the Jsoup JavaDocs there are tons of options to customize the exact request and response you wish to work with. And you can easily replicate HTTP-connectors Input/Output abilities with some JS functions that look for Input parameters on the task.

See: Nashorn extensions - Nashorn extensions - OpenJDK Wiki for further details about all the ways you can import packages and classes with Nashorn. In this case we are using the with() method that allows us to encapsulate the package in its own scope and not pollute the global scope.

The one item that would be great from @camunda is the ability to execute scripts as part of Service Tasks, Send Tasks, and Throw Message Events so that the BPMN does not have to be script tasks everywhere (or not having to use Execution events / input/output parameters).

Some quick examples of really useful methods from Jsoup:



A few notes on reasons for things:

  1. the spinify method is used because in order to properly stingify the json response we need to parse it, otherwise the stringify will add /n characters for pretty printed json responses
  2. ignoreContentType() is used int he .connect() method because Jsoup does a pre-parsing content type validation check that looks for the pre-approve content types that Jsoup is optimized for (text, html, xhtml, xml, etc (HTML related content types))
  3. The .execute() method is used instead of get(), post(), etc because the execute() method does not parse the response for html.
  4. Take a look at the many different ways to create headers and data. Maps are supported, query params, Form Data, etc
  5. Note the enum usage in the .method() line. The enums are a specific type and because JS is typeless you need to force the type with Java.type(). See the Nashorn extensions - Nashorn extensions - OpenJDK Wiki docs for more info.
  6. Another great benefit is that Jsoup provides much more robust errors/exceptions that provide better Catch() capabilities and just general debugging.

Would love to hear anyones thoughts on this approach. So far our testing is showing very fast execution times and other than a patterns to develop (such as the spinify() method, and the usage of ignoreContentType() in the code sample above), everything “just works” as one would expect with a HTTP request DSL.

Using http-connector to access oAuth Secured APIs
ScriptTask: perform a http method?
Http Connector Post Method With Content-Type application/x-www-form-urlencoded
Catch Service not available exception
Notify task completion using javascript in execution listener
Handle technical problem in http-connector
Raising an incident via inline script vs incident generated by exception
HTTP / HTTPS Proxy & http-connector
How to use Basic authentication with Connector
Call REST API from listener
[Connector] Configuring properties
Http-connector ConnectionTimeout
Connector HTTP throw BPMN error
Yes Again more problems with HTTP Connector just stuck
Deadlocked Processes, maybe
Intermittent stalling on event with listener
File Typed Value Performance vs File storage on external source
Rest Api based workflow - Http-Connector and handeling errors
Using Freemarker for dynamic header in camunda connect
How to upload documents via the camunda connector?
Set timeout for http-connector service task
Call REST API via service task instead of using http connector
How to convert an http-connector service task to an external task
Dynamic http-connector header
Process instance hung after camunda restart
Using inline Javascript listener
Send processInstanceId with HTTP-connector service task
Javascript HTTPS POST in Listener
How to make HTTPS call
Http-connector Rest Dynamic url configuration
Http Connector vs Java Service Delegation
Updating a connector in a Java Delegate during execution
Custom http-connector?
Can HTTP-Connector Receive Binary Data?
Send SMS through Service Task (http-connector) and Twilio (Script Based)

For anyone interested in web scraping html:

you can also import html into a Spin XML object and then use SPIN’s xPath capabilities to parse the html.

Something like this:

function getUrlAsXhtmlString(url)
  with (new JavaImporter(org.jsoup))
    var doc = Jsoup.connect(url).get();

    var docString = doc.html();

    return docString;

function generateSpinVariables(xHtmlString)
  var htmlSpin = S(xHtmlString);
  return htmlSpin;

function scrape(url)
  var xHtmlString = getUrlAsXhtmlString(url);
  return generateSpinVariables(xHtmlString);

var xhtml = scrape('http://myurl');

var links = xhtml.xPath('//main//ul/li/a/@href').attributeList();

1 Like

Here is another interesting use case where you could use Jsoup + variablesInResult parameter of the /start process-definition endpoint to provide a validations workflow.

Example: you submit a form, and you need to have server side validations with other systems.

Could also be 4 sequential scripts. The performance was the same.

So your setup would look something like:

The function.js looks like this:

if (execution.hasVariable('valueToValidate')){
  var valueToValidate = execution.getVariable('valueToValidate')
} else {
  throw 'valueToValidate variable does not exist'

with (new JavaImporter(org.jsoup))
  var doc = Jsoup.connect('')
                  .header('Accept', 'application/json')
                  .data('filterABC', valueToValidate)

  var resBody = doc.body()

function spinify(body)
  var parsed = JSON.parse(body)
  var stringified = JSON.stringify(parsed)
  var spin = S(stringified)
  return spin

execution.setVariable('response', spinify(resBody))
// execution.setVariable('response', resBody)

The performance was about the same for returning a SPIN json object vs a string.


Stephen, thank you so much for all your sharings of real examples and how to’s!

I want to add my two cents :slight_smile: If you need only simple API call from scripting task, you may reach that by calling internal HTTP connector as shown below. In this case you do not have to add any library to classpath.

in JavaScript script task:

var httpConnector = org.camunda.connect.Connectors.http()
var resp = httpConnector.createRequest()

var result = resp.getResponse()

execution.setVariable('responseBodyString', result)

Here we call Java HTTP connector class from JS, so we might use the scenario from official docs for POST, add headers and so on:



@dmitrysd great example! and it fills in a usage example from a while back about using HTTP Connector directly through scripting.

Two issues come to mind to be aware of when using this solution:

  1. You will still have the timeout issue that is discussed above and in links: HTTP Connector does not have a built in Timeout and thus can create jobs that last “forever”, which can result in Job Executor blocking (executor is stuck executing jobs that will never end)

  2. HTTP Connector cannot read binary data: so you will not be able to download binary files as they will be converted to Strings and data will be lossed. Again this is discussed in other pages that discuss HTTP Connector handling of binary data such as downloading PDFs.


Thanks for the sharing. I am now using Camunda 7.9. Is there a guide for how to install Camunda with Jsoup?


@liang you must add it to the class path similar to: See the dockerfile. Then you can access it like any other class


Hi, @StephenOTT. Thank you for this lib, it works.
But how can i handle timeout (exception) from Jsoup request not as error for Camunda? In my case there is no needs to retry if timeout

@raliev you use Jsoups timeout feature/method and you wrap your Jsoup request in a try/catch

Oh, of course! Thanks!! Can i do it with nashorn JS script too?

Yes. There is a specific exception from Jsoup that is related to a timeout.

1 Like


@StephenOTT its possible to use jsoup without docker ? Can i add the “jsoup.jar” to “camunda-bpm-tomcat-xxx/server/apache-tomcat-xxx/lib” folder and use it directly?


Yes you can. If you are not using docker, you may just want to add the Jsoup dependency into your build.

1 Like

@StephenOTT i am using jsoup and making both get and post calls in camunda v7.14- working absolutely fine.
But the same flow is not working in camunda v7.15, with the below error:
TypeError: (((Jsoup.connect(custom_url).method(Java.type(“org.jsoup.Connection.Method”).POST)).header(“Content-Type”, “application/json”)).header(“Authorization”, authbearer)).requestBody is not a function i

**Note: I am using same version of jsoup jar - 1.60 and 1.13.1
and have placed the jars in the engines of both camunda v7.14 and v7.15
Can you guide me here, thank you

Do you have the ability to throw together a quick unit test ? Just a little faster for me to test. If not, no issues. But will check it out tomorrow.

I don’t have any such unit test in place, please take your time. No issues.

ok it was the issue with version only,
After upgrading to jsoup to 1.13 restarted tomcat.
I was able to use thank you.

Okay glad to hear it is working

@StephenOTT @teja_polisetty

I am using camunda-tomcat-7.16 and replaced jsoup-1.14.3.jar to camunda-bpm-tomcat-7.16.0\server\apache-tomcat-9.0.52\lib, But when i am running the script it is throwing me below error,

Cannot instantiate process definition Process_0my27p5:3:e08e6d5a-921d-11ec-b642-7c7a9180cd61: Unable to evaluate script while executing activity ‘Activity_1uu11sx’ in the process definition with id ‘Process_0my27p5:3:e08e6d5a-921d-11ec-b642-7c7a9180cd61’:org.graalvm.polyglot.PolyglotException: ReferenceError: JavaImporter is not defined

can you provide any inputs on this!


That is a graal error related to the polyglot js engine. If you remove jsoup does the error disappear?