JSON and Nullable elements

The nullable JSON issue

Using DataFlex it is very convenient to fill up your JSON data from a struct when using the cJSONOjbect class. See for example our Create JSON from struct article.

Quite often that JSON data is then send to a REST service and depending on that service they might not want to receive all the elements in your JSON in order to work correctly.

There's also the issue of receiving JSON data with a null in there and not being able to process that automatically via a struct as DataFlex variables are not nullable.

This has been debated before in the forum (1) a few times and currently the official way to do so is by traversing the JSON data structure and then removing the members that the other side does not want to process by using the RemoveMember method.

  Send RemoveMember of hoJSON "myElement"

Please note that when you use JsonToDataType to go from JSON to a struct that you now have to turn off the strict checking if all members are available with

  Set pbRequireAllMembers of hoJSON to False

That works fine for simple JSON structures. If your structure has another level then your code becomes more difficult to write pretty fast.

Take for example the following simplified JSON

{
 "type": "Sales",
 "companyCode": "DEMO",
 "date": "2013-06-29",
 "customerCode": "ABC",
 "addresses": {
   "shipTo": {
     "line1": "143 Main Street",
     "city": "Irvine",
     "region": "CA",
     "country": "US",
     "postalCode": "92614",
     "longitude": 0.0,
     "latitude": 0.0
   },
   "shipFrom": {
     ... more details ...
   }
 }}

If that's filled in from a struct and your REST service gets confused if you pass both address as well as long/lat coordinates with value 0.0 then you have to remove the longitude/latitude elements.

To do this your code could look like:

..
      Get Create (RefClass(cJsonObject)) to hoJsonRequest
      If (hoJsonRequest) Begin
        Send DataTypeToJson Of hoJsonRequest data
        If (data.addresses.shipto.latitude=0.0 and data.addresses.shipto.longitude=0.0) Begin
          Send RemoveAddressCoordinates hoJsonRequest "shipTo" "latitude"
          Send RemoveAddressCoordinates hoJsonRequest "shipTo" "longitude"
        End
..

with the RemoveAddressCoordinates method looking like this:

  //
  // Used to strip longitude/latitude data from json
  //
  Procedure RemoveAddressCoordinates Handle hoJsonRequest String sAddressType String sCoord
    Integer eType
    Handle  hoAddresses hoAddressType
    
    Get MemberJsonType of hoJsonRequest "addresses" to eType
    If (eType=jsonTypeObject) Begin
      Get Member of hoJsonRequest "addresses" to hoAddresses
      If (hoAddresses) Begin
        Get MemberJsonType of hoAddresses sAddressType to eType  // "shipTo"
        If (eType=jsonTypeObject) Begin
          Get Member Of hoAddresses sAddressType To hoAddressType
          If (hoAddressType) Begin
            Send RemoveMember Of hoAddressType sCoord // "latitude"
            Send Destroy Of hoAddressType
          End
        End
        Send Destroy Of hoAddresses
      End
    End
  End_Procedure

Yes it's readable, No it's not so easy to support once you get multiple levels deep for your address element as you now have to make sure you destroy the objects etc..

The RemoveNamedMember method

So when looking at the line with the actual condition on when to remove a specific element :

  If (data.addresses.shipto.latitude=0.0 and data.addresses.shipto.longitude=0.0) Begin

it looked like it would be good to have a method that could take care of this in one line as well.

eg. this would be nice to have:

  If (data.addresses.shipto.latitude=0.0 and data.addresses.shipto.longitude=0.0) Begin
    Get RemoveNamedMember hoJsonRequest "addresses.shipTo.latitude"
    Get RemoveNamedMember hoJsonRequest "addresses.shipTo.longitude"
  End

it will then also work for a json structure where the address is a level deeper without having to write new code.

eg.

    Get RemoveNamedMember hoJsonRequest "Returns.addresses.shipTo.longitude"

Turned out it was fairly easy to write (easier than writing this post!)

  
  //
  // Used to strip a named/value pair from json
  // You can use this to directly remove a JSON member at a lower level from the JSON
  // object passed via hoJSON.
  // The member to be removed uses the exact JSON member names separated by dots.
  // Beware that JSON member names are case sensitive!
  //
  // If the member does not exist a runtime error will be triggered.
  // Eg.
  //  Send RemoveNamedMember hoJsonRequest "Order.addresses.shipTo.latitude"
  //
  Procedure RemoveNamedMember Handle hoJson String sName
     Integer iDotPos
     Integer eType
     Handle  hoChild
     String  sMember
     
     Move (Pos(".",sName)) to iDotPos
     If (iDotPos>0) Begin
       Move (Left(sName,iDotPos-1)) To sMember
       Move (Replace(sMember+".",sName,"")) To sName
       //Get HasMember of hoJson sMember to bHasMember <-- use this if you want to filter the runtime error (we currently do not)
       Get MemberJsonType of hoJson sMember to eType
       If (eType=jsonTypeObject) Begin
         Get Member of hoJson sMember to hoChild
         If (hoChild) Begin
           Send RemoveNamedMember hoChild sName
           Send Destroy Of hoChild
         End
       End
     End
     Else Begin
       Send RemoveMember Of hoJson sName
     End
  End_Procedure

Note that if you don't want the runtime error that you can use the HasMember test.

We don't use that as we always first convert from struct so we know the element exists. If the element is not there then there's likely a case sensitivity issue, so getting a runtime error helps when debugging your code.

Of course a little while later I bump into a more complicated issue where the removed member is in an array. Lucky for me, in an single dimension array. This can be taken care of like this:

  
  //
  // Used to strip longitude/latitude data from json in a CreateTransactionModel
  // You can use it to directly remove a JSON member at a lower level from the JSON
  // object passed via hoJSON.
  // The member to be removed uses the exact JSON member names separated by dots and
  // this is case sensitive.
  //
  // If the member does not exist a runtime error will be triggered.
  // Eg.
  //  Send RemoveNamedMember hoJsonRequest "newTransaction.addresses[2].shipTo.latitude"
  //
  Procedure RemoveNamedMember Handle hoJson String sName
     Integer iDotPos
     Integer iLSBPos iIndex
     Integer eType
     Handle  hoChild
     Handle  hoArray
     String  sMember sIndex
     
     Move (Pos(".",sName)) to iDotPos
     If (iDotPos>0) Begin
       Move (Left(sName,iDotPos-1)) To sMember
       Move (Replace(sMember+".",sName,"")) To sName
       Move (Pos("[",sMember)) To iLSBPos
       If (iLSBPos>0) Begin
         Move (Right(sMember,(length(sMember)+1)-iLSBPos)) To sIndex
         Move (Replace(sIndex,sMember,"")) To sMember
         Move (Replace("[",sIndex,"")) To sIndex
         Move (Replace("]",sIndex,"")) To sIndex
         Move (Cast(sIndex,Integer))   To iIndex
       End
       //Get HasMember of hoJson sMember to bHasMember <-- use this if you want to filter the runtime error (we currently do not)
       Get MemberJsonType of hoJson sMember to eType
       If (eType=jsonTypeObject) Begin
         Get Member of hoJson sMember to hoChild
         If (hoChild) Begin
           Send RemoveNamedMember hoChild sName
           Send Destroy Of hoChild
         End
       End
       Else If (eType=jsonTypeArray) Begin
         Get Member of hoJson sMember to hoArray
         If (hoArray) Begin
           Get MemberByIndex of hoArray iIndex to hoChild
           If (hoChild) Begin
             Send RemoveNamedMember hoChild sName
             Send Destroy Of hoChild
           End
           Send Destroy Of hoArray
         End
       End
     End
     Else Begin
       Send RemoveMember Of hoJson sName
     End
  End_Procedure

Harm's RemoveEmptyMembers method

Harm Wibier posted a solution where you can automatically remove elements which have a value of 0 or in the case of a string element where the string element is an empty string. See also (2)

//
//  Recursive function going over JSON objects removing all empty strings, 0 and NULL values of objects (not from arrays).
//
//  Params:
//      hoJsonObj       Handle of the JSON object to process.
//      bEmptyString    If true empty string values will be removed.
//      bZeroNumber     If true all numeric 0 values will be removed.
//      bNull           If true all NULL values will be removed.
//
Procedure RemoveEmptyMembers Handle hoJsonObj Boolean bEmptyString Boolean bZeroNumber Boolean bNull
    Integer iTo iMember iType
    Handle hoMember
    String sMemberName
    Boolean bRemove
    
    Get MemberCount of hoJsonObj to iTo
    For iMember from 0 to (iTo - 1)
        Move False to bRemove
        
        Get MemberByIndex of hoJsonObj iMember to hoMember
        
        Get JsonType of hoMember to iType
        
        Case Begin
            Case (iType = jsonTypeString)
                If (bEmptyString) Begin
                    Move (JsonValue(hoMember) = "") to bRemove
                End
                Case Break
            Case (iType = jsonTypeInteger)
                If (bZeroNumber) Begin
                    Move (JsonValue(hoMember) = 0) to bRemove
                End
                Case Break
            Case (iType = jsonTypeNull) 
                Move bNull to bRemove
                Case Break
            Case (iType = jsonTypeObject or iType = jsonTypeArray) Begin
                Send RemoveEmptyMembers hoMember bEmptyString bZeroNumber bNull
                Case Break
            End
        Case End
        
        //  Only remove empty members of objects (not from arrays)
        If (bRemove and IsOfJsonType(hoJsonObj, jsonTypeObject)) Begin
            Get MemberNameByIndex of hoJsonObj iMember to sMemberName
            Send RemoveMember of hoJsonObj sMemberName
            Decrement iMember
            Decrement iTo
        End
        
        Send Destroy of hoMember
    Loop
End_Procedure

// Use like this:
Send RemoveEmptyMembers hoJson False False True

That works, but I'm personally a bit cautious about removing all elements that have value "0". Sometimes we want pass that actual value, empty or zero does not equal null. There might be another element with the value 0 that should not be removed. However, sometimes it is exactly what you need.

If you just want to remove the nulls before you move the data to a struct, then perhaps Mike Peat's variant of Harm's code is a bit more clear:

// Version of Harm's procedure, just for nulls.
Procedure RemoveNullMembers Handle hoJsonObj
    Integer iTo iMember iType
    Handle hoMember
    String sMemberName
    Boolean bRemove
    
    Get MemberCount of hoJsonObj to iTo
    Decrement iTo

    For iMember from 0 to iTo
        Move False to bRemove
        Get MemberByIndex of hoJsonObj iMember to hoMember
        Get JsonType of hoMember to iType
         
        If (iType = jsonTypeNull) ;
                Move True to bRemove
        Else If (iType = jsonTypeObject or iType = jsonTypeArray) ;
                Send RemoveNullMembers hoMember
         
        If bRemove Begin

            If (IsOfJsonType(hoJsonObj, jsonTypeObject)) Begin
                Get MemberNameByIndex of hoJsonObj iMember to sMemberName
                Send RemoveMember of hoJsonObj sMemberName
                Decrement iMember
                Decrement iTo
            End
            Else If (IsOfJsonType(hoJsonObj, jsonTypeArray)) Begin
                Send RemoveMember of hoJsonObj iMember
                Decrement iMember
                Decrement iTo
            End

        End
         
        Send Destroy of hoMember
    Loop

End_Procedure

See also (3)

External references

• (1) YAFR DataTypeToJson beyond 19.1 loong and mostly off topic debate at the forum about null support and JSON
• (2) JsonToDataType Harm Wibier's solution
• (3) Json-Nulls Mike Peat his variant
• RemoveNulls function by Mike Peat