How to properly de-/serialize Collections to JSON

Camunda 7 Topics Discussion & Questions
1

Hi folks,

I use the Spin plugin to serialize process variables to JSON. This works fine in cases, where I put a single entity into the variables, its Object Type Name will be the correct one.

However, when I put a Collection, let’s say a HashSet of my entity into the Variables, the Object Type Name will always be java.util.HashSet<java.lang.Object>, regardless of the actual type.

When I put the Collection and retrieve it again in the same execution, it will be taken from the cache and deserialization works fine, but when it is not taken from the cache, it is deserialized into a HashSet, which contains a LinkedHashMap, which again contains the content in key-value-pairs.

Is there any way to properly de-/serialize Collections without loosing the knowledge of the actual type?

Cheers,
Stefan

2

I did not use spin for this, but maybe it helps: When I had to serialize a List in a process variable, I created a concrete type for this, so instead of List I used a CustomerList.
Using guava, this can be easily done by using ForwardingList. Make the custom type serializable, and you should be good to go.

3

Where you thinking of managing the collections as JSON documents - loosely typed and without schema?

The type remains consistent. Though, the object itself maintains this as a sort-of implied expectation. Enforcing schema (like XSD for XML), for me at least, implies overhead and additional maintenance - just another bit of extra code. I prefer to verify form as a method of checking for necessary information rather than entire document (JSON) itself.

I’ve been using JSON documents as JSON (Jackson JsonNode) - no underlying object-type required. In-other-words, I do not send the entire JSON document through a deserializer as a pre-condition of use/reference.

However, SPIN seems to work in that it maintains very detailed information with regards to originating source-object. Sometimes… it appears this is too detailed. But, I’m simply after the enclosed information, which enforces type/schema via logical expectations as to where things are in context to self.

4

If it is just a serializable POJO, will it be somehow readable in Cockpit? That’s why I want to use Spin and JSON, to get readable content in Cockpit.

Unfortunately, if I create a custom type, that wraps a Set<FullBlownEntity>, I have to create a custom deserializier as well, otherwise it does not seem to work.

I tried another approach, and instead of putting a Set<FullBlownEntity> to my variable, I just put a Set<Long> containing the IDs. But even there, when I deserialize it later, I get it back as Set<Integer>.

5

If I get your idea correctly, I have to map the stuff that want to put into the variables into JSON myself? I could certainly do that, but by using spin I wanted to get rid of this custom JSON mapping code throughout all my delegates…

6

I’ll try to shed some light what is going on.

When you set a collection variable to be serialized as JSON via Spin, type information must be persisted along with the JSON. That means: What is the type of the collection? What is the type of each element (and possibly the types of any referenced objects)? If this is not known, it is impossible to deserialize JSON to Java objects.

Now, there are several ways to do that with certain tradeoffs. Since Spin uses Jackson, let’s have a look at Jackson’s way to treat polymorphic (de-)serialization. Have a look at the Jackson docs: https://github.com/FasterXML/jackson-docs/wiki/JacksonPolymorphicDeserialization. In particular, the recommended approach is to include type information in the JSON itself via additional metaattributes (there are various ways to do that, explained in the document linked above).

This is not how Spin does it though. Instead, Spin stores type information separately in a different database field. It uses a Jackson method to determine a canonical type name for an object. This type name tells Jackson during deserialization what the collection type is and what the element type is. For instances of java.util.List, there is also a custom extension that would deduct the element type from the class of the first element in the list. Relevant code is in this package: https://github.com/camunda/camunda-spin/tree/master/dataformat-json-jackson/src/main/java/org/camunda/spin/impl/json/jackson/format

This keeps the JSON nice and clean, especially when used outside of a Java environment. Of course, this approach has certain drawbacks, because what can be expressed with these type names is rather limited. For example, for your Set<FullBlownEntity, the type name becomes java.util.HashSet<java.lang.Object> which is not enough to deserialize the elements from JSON to FullBlownEntity, so Jackson falls back to its default and deserializes Json to java.util.Map (here LinkedHashMap apparently). For Set<Long> the behavior is the same, but Jackson’s fallback for JSON numbers is to create instances of Integer (or probably Long in case they exceed the Java integer range). And lastly, I learned a couple of months ago that the way Spin uses these canonical type names is not an official Jackson feature.

So in the end, the fail-safe way for polymorphic object deserialization would be to go with how it is supposed to be handled in Jackson, i.e. with extra meta attributes in the JSON itself. To enable that, you will have to configure the ObjectMapper Spin uses. See the following example on how to configure the object mapper: https://github.com/camunda/camunda-bpm-examples/tree/master/spin/dataformat-configuration-global

Hope that helps.

Cheers,
Thorben

1 Like
7

Thanks for the details -

Mind if I add in a question or comment?

In using SPIN, I’d prefer to just simply copy (deepcopy) a node into the Camunda process variable. Because, once managed as a Camunda JSON type, access to fields and lists/arrays dovetails into the form SDK.

Noting that I added a ReST, JAX-RS facade for Camunda Java-API access…

I’m checking these samples into: github
NOTE: ideally working towards a CMIS adapter… documented or data oriented BPM/Case

Starting with a JsonNode postpayload
public JsonNode caseBasicStartSpin(JsonNode postpayload)

This is the preferred, low-code approach:

variables.put(jsonNode.get("name").asText(), Spin.JSON(jsonNode.get("value").deepCopy()));

Problem was that it (above) created a JsonNode with way too much detail… Referred to in your above response, “extra meta attributes in the JSON itself”.

So, I used this method which required a “writeValueAsTring” via Jackson mapper:

variables.put(jsonNode.get("name").asText(), Spin.JSON(mapper.writer().writeValueAsString(jsonNode.get("value"))));

Problem with this approach is that I didn’t like having to write the whole thing back to a string. But, at least I wasn’t bothered with tedious character escapes.

I’m guessing that, as JSON documents (schema-free) came into their own domain, the whole object type association is a thing-of-the-past. For example, why bother with Java types when all we really want is the document itself. Reason driving this point is that I want to avoid maintaining Java classes when all I want are JSON documents.

8

In my example, I do use the Camunda SPIN type - so far working. Currently adding integration with Camunda’s form SDK. Though, I ran into a few annoyances with Eclipse Neon v3 editor…

Seems that the extra “<script” attributes aren’t copasetic with Neon’s HTML view.
This breaks the Eclipse HTML/script editor goodness:

<script cam-script type="text/form-script">

see end of this thread for some code examples

Also, for gitHub, see method:

public JsonNode caseBasicStartSpin(JsonNode postpayload)"

Also, take a look at how I verify process variables with a “print” method delegate:

NOTE: with Camunda Case, there’s a different delegate type - remember to use the correct one per model -

BPMN:

public void printvariables(DelegateExecution execution)

CMMN/Case:

public void printCaseVariables(DelegateCaseExecution execution)
9

Thank you Thorben!

I would be fine with polymorphic serialization, even thoutg it kind of pollutes my JSON with meta information.

I tried to configure the ObjectMapper, used by Spin, to use GlobalDefaultTyping, but it would only add meta information ffor the first level of custom classes, e.g. my FullBlownEntity. If my FullBlownEntity has a property of type AnotherFullBlownEntity, meta information is missing, which causes again issues when deserializing.

I will keep looking into this, at least I think you pointed me into the right direction.

10