Using/building a "webhook server" for Post /message

Looking at a generic way to capture messages to send to camunda without exposing to Camunda API.

I through together a small ruby/sinatra/redis example that allows Camunda to generate a record in Redis (key is a UUID) and then when a system wants to respond to that message they have to know the UUID, if the UUID is found in the Redis DB, then the message is passed to Camunda. If the message is successfully received by camunda, then the record in Redis is deleted.

Anyone ever built or use something similar?

in this simple example the goal was to not have any form of authentication. If the system knows the UUID they can send the message to camunda.

Hi Stephen,
I used the same pattern once however I used aws Dynamo. My use case was the same, Dynamo absorbs any brute force attacks and gate keeps legitimate requests…another pattern I considered was not only use a uuid as a token, but add an Hmac such that the token has a restricted context…

Rob

@Webcyberrob you still have use case? Was also thinking this is a easy use case for lambda serverless app.

I was also was looking at adding hmac. I was using a GitHub we hook Sinatra app as a template which supported hmac auth.

Any lessons learned to share?

Hi Stephen,

Some additional info.
I used the UUID as the key.
I stored additional parameters in the DynamoDB against the key.
My use cases were things like pay a bill. Customer had choice of pay now, pay when due, Pay with instrument x etc. Hence I could preconfigure actions and just expose them as a single UUID. Thus no parameters could be tampered with to under or overpay a bill for example.

Main challenges were distributed transactions without a transaction manager. Hence I had think carefully about idempotency and eventual consistency.

You may need a garbage collecttor in your UUID store. Hence you may want to store a Time To Live (TTL) and use a reaper process to clean up…

R

1 Like

Great info. Thanks @Webcyberrob

@Webcyberrob my other use case that I was thinking about is that using the UUID you could create URLs (with TTL) that allow users to respond to workflows through Email and other Non-Tasklist / Non-authenticated methods. Example a manager selecting a Approve or Deny to an action/content directly in a email. The link would be the URL with the UUID and the parameters for the Approve/Deny response. Obviously this would have some extra layers to ensure that someone cannot just inject process variables and other potentially malicious information.

Other use cases that come to mind:

  1. Responses directly in Email
  2. Responses from other channels such as sms, bots, etc.
  3. Responses that are embedded in other websites or websites that are short term with TTL decision or multi-decision points. It would be up to the client to manage the experience, and send the data to the webhook. The webhook would manage/parse the response based on the special conditions in place for that type of request. (Example a Email Approve / Deny would only allow a Approve or Deny response, anything else would be ignored or denied, and the user would have to try again).

@Webcyberrob

Main challenges were distributed transactions without a transaction manager. Hence I had think carefully about idempotency and eventual consistency.

Why was idempotency an issue in your use case/design? In my thinking of this issue, I would have locked the UUID record while the request is sent to Camunda. If camunda is successful then UUID record is deleted, if camunda fails, then UUID is unlocked. If attempt is made and UUID is locked, then the webhook server would deny the requestor.

In my use cases I mainly think of the messages as Unique. So a message is only for a certain point in time. If more than system is going to send messages to camunda using the same message, then multiple UUID records would be created (at least in my thinking of the scenario(s)).

Thoughts?

Think of the client use case, particularly on a lost response from the server. The client does not know if the message was processed or not. If the client retries and you have already deleted the UUID, then the client gets an error. Now the client does not know if their message was already processed, or it was an invalid token in the first place…

The same applies between your UUID store and Camunda Engine, think of a lost response from the Camunda engine, was the message received or not? If you retry, and the message was already correlated, you will get an error etc. BTW, I had a pattern for making message correlation idempotent in the engine. AFter successful correlation, have the process enter a new scope (inline sub-process). Inside that scope, use a non interrupting event subprocess to passively consume the event. Hence on a lost response, if a client retries, you will have somewhat of an idempotent behaviour…

regards

Rob

Ahh interesting use case!!

So at a high level, the core issue i think would be resolved (in my variation) if the UUID record’s value was a json string additionally contained a status. That status would be set to something like “completed” when the message was successfully received. When the client would attempt to send the same message, they would receive the “message already sent” error response.

But i really like to think about your use case!
Would also like to hear more about your payment scenario if you can share.

Hi Stephen,

My use case was similar you your ideas. I wanted the concept of a ‘one click’ user experience. As another use case, lets assume the user has a prepaid debit card which can be topped up from a registered bank account. When the prepaid balance is low, I initiate a topup process. The process can generate say 3 actions, topup by $20, $50, Other Amount… Hence the topup by $20 or $50 I can encode in paraemters linked to the UUID. Hence customer presses the link in the UI, the UUID conveys the intent, no data entry required (if its the pre-canned amount). In addition, we can learn over time what the user’s preference is and thus the system adapts and self-tunes to the user.

I can use a similar approach to bill payment. There may be three actions, Pay Now, Schedule Payment,Remind Me. Hence each action linked to a UUID, one click user experience. If the bill was paid out of band, then my bill payment process can react and effectively revoke these UUIDs. If the bill becomes overdue, then the remind me option and the schedule payment options can be revoked.

Hence I used BPMN to manage the customer experience, I used CQRS as the architectural style to enable scale, I realised an adpative, one click user expereince and I used these one time use tokens to enable a security model where as little information as possible is exposed and denial of resource attacks are managed in the right layer…

In modelling the user experience in BPMN, I found it was better to use an FSM type paradigm. Consider the use cases above, the bill is due, paid or overdue. A registered payment instrument may be current, revoked, replaced etc. Hence the user experience ideally taps into the lifecycle events of all the resources in its domain and reacts accordingly. If the customer revokes a registered payment instrument, then the bill process experience should revoke the schedule or scheduled payment options etc…

regards

Rob

1 Like

@Webcyberrob you rock!

1 Like

Had this use case come up today in some example scenarios:

I whipped together a hapi/node server + the twillio example to send “approve/deny” URLs to the user.

Just posting for anyone that comes across this use case later.

Node Code

'use strict';

const Hapi = require('hapi');
const Boom = require('boom');
const loki = require('lokijs');
const Wreck = require('wreck');
const uuidV4 = require('uuid/v4');

var db = new loki('loki.json')
var decisions = db.addCollection('decisions')

const server = new Hapi.Server();
server.connection({ port: 3000, host: 'localhost' });


// Create a Decision UUID.
server.route({
  method: 'GET',
  path: '/decisions',
  handler: function (request, reply) {
    var createdDecision = decisions.insert({"decisionid" : uuidV4() });
    reply({"decisionid" : createdDecision.decisionid});
}}) // End of Route

// Create a Decision Outcome (Approved/Denied)
server.route({
  method: 'GET',
  path: '/makedecision/{decisionid}',
  handler: function (request, reply) {

  var decisionid = request.params.decisionid;
  var decision = request.query.decision;

  // Check if the decision already exists.
  if (decisions.find({'decisionid': { '$eq' : decisionid }}).length == 0 ){
    return reply(Boom.badRequest('Incorrect decision id'););
  }

  if (decision != "approved" && decision != "denied"){
    return reply(Boom.badRequest('Invalid decision query param')); 
  }

  // Builds the data needed by the Camunda /messages endpoint
  var camundaURL = "http://192.168.99.100:8080/engine-rest/message"
  var camundaPayload = {
      "messageName" : "decision",
      "resultEnabled" : true,
      "correlationKeys" : {
        "decisionID" : {"value" : String(decisionid), "type": "String"}
      },
      "processVariables" : {
        "decisionOutcome" : {"value" : decision, "type" : "String"}
      }
    };

  //POST /Message to Camunda
  Wreck.post(camundaURL, { "payload" : camundaPayload, "json" : true }, (err, res, payload) => {
        reply({"response" : payload });
  });

}}) // End of route

server.start((err) => {

    if (err) {
        throw err;
    }
    console.log(`Server running at: ${server.info.uri}`);
});

BPMN:
uniqueURL-template.bpmn (17.1 KB)

Notes:

  1. very little is error caught or bullet proof. This was just a proof of concept to demonstrate a quick example.
  2. Consider this a starting point to extend from.
  3. Decision UUIDs are not persisted.
  4. Decision UUIDs are stored in-memory db (lokijs)
1 Like