Nifi - How to add key:value to json

[#1]

Hi, we need add a default key:value with actual timestamp to the following json input structure:

[ {
  "host" : "DUSTSADMIN.ads.xyz.de",
  "host_icons" : "menu",
  "host_state" : "UP",
  "num_services_crit" : "0",
  "num_services_ok" : "28",
  "num_services_pending" : "0",
  "num_services_unknown" : "0",
  "num_services_warn" : "0"
}, {
  "host" : "DUSTSVMDC01.ads.xyz.de",
  "host_icons" : "menu",
  "host_state" : "UP",
  "num_services_crit" : "0",
  "num_services_ok" : "34",
  "num_services_pending" : "0",
  "num_services_unknown" : "0",
  "num_services_warn" : "0"
}, {
  "host" : "DUSTSVMDC02.ads.xyz.de",
  "host_icons" : "menu",
  "host_state" : "UP",
  "num_services_crit" : "0",
  "num_services_ok" : "34",
  "num_services_pending" : "0",
  "num_services_unknown" : "0",
  "num_services_warn" : "0"
} ]

needed output:

[ { "ctime" : "Current_timestamp",
  "host" : "DUSTSADMIN.ads.xyz.de",
  "host_icons" : "menu",
  "host_state" : "UP",
  "num_services_crit" : "0",
  "num_services_ok" : "28",
  "num_services_pending" : "0",
  "num_services_unknown" : "0",
  "num_services_warn" : "0"
}, {"ctime" : "Current_timestamp",
  "host" : "DUSTSVMDC01.ads.xyz.de",
  "host_icons" : "menu",
  "host_state" : "UP",
  "num_services_crit" : "0",
  "num_services_ok" : "34",
  "num_services_pending" : "0",
  "num_services_unknown" : "0",
  "num_services_warn" : "0"
}, {"ctime" : "Current_timestamp",
  "host" : "DUSTSVMDC02.ads.xyz.de",
  "host_icons" : "menu",
  "host_state" : "UP",
  "num_services_crit" : "0",
  "num_services_ok" : "34",
  "num_services_pending" : "0",
  "num_services_unknown" : "0",
  "num_services_warn" : "0"
} ]

[#2]

One way to do this is to first use the EvaluateJsonPath processor and break out the json into individual attributes. Notice that you can add ctime by clicking the + symbol, adding a name and a value.

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.2.0/org.apache...

See image, evaluatejsonpath, below.

Then later, you can use the AttributesToJson processor to rebuild the complete JSON event.

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.2.0/org.apache...

See image, attributestojson, below.The output at this point would look like this

{
  "twitterMessage" : "20x faster VM access w/ Dell EMC and Intel. Learn more  https://t.co/1hD0MYfBbh https://t.co/0GhsMQve8b",
  "twitterUser" : "GCSIT Solutions",
  "ctime" : "Wed Jun 07 18:10:35 +0000 2017"
}

[#3]

In addition to @bhagan's answer, as of NiFi 1.2.0 (due to NIFI-3010), you can use NiFi Expression Language in JOLT specifications, so with a JoltTransformJSON processor you could have the following Chain spec:

[{
  "operation": "default",
  "spec": {
    "*": {
      "ctime": "${now()}"
    }
  }
}]

If you want the ctime field to contain a string with the number of milliseconds since Epoch, you can use now():toNumber() instead of just now(). If you want an actual numeric value (rather than a String containing the numeric value), you need an additional operation to change the value to a long:

[{
  "operation": "default",
  "spec": {
    "*": {
      "ctime": "${now():toNumber()}"
    }
  }
}, {
  "operation": "modify-overwrite-beta",
  "spec": {
    "*": {
      "ctime": "=toLong"
    }
  }
}]

[#4]

@Matt Burgess, Love JOLT transformation solution, we use it a lot with dynamic jsons, transpose, etc... but in this case, I would go simple, just replace text. let me know if I miss something (check my alternative)

[#5]

Hi @Matt Burgess.

First I want do say thank you for your work!

Second, could you please update your answer. The first part about adding an element works like charm, but when I add the part about modifying string to long the JoltTransformJSON processor thorws an error.

\" " is invalid because Specification not valid for the selected transformation \"

I'm using Nifi 1.4.0

[#6]

this worked for me so easily.. i tried the replacetext suggested below.. but it replaced every {. Jolt seems like the clean and easy way to do this. Kudos @Matt Burgess

[#7]

@Timo Burmeister,

Keep it simple:

use replaceText processor with this configuration:

Search Value         :  [{]
Replacement Value    :  { "ctime":"${now()}",
Replacement Strategy :  Regex Replace
Evaluation Mode      :  Entire text

[#8]

True, if you are just inserting a single new field at the top-level of a flat JSON doc, this should work fine. However you may want an optional newline between the array and the first object if the JSON is pretty printed. Also, would this work if you had a nested array of objects further down in the file? I guess you could put in a start-of-line into the regex, but if the JSON has weird whitespacing you may run into the same problem. I recommend JOLT in general because it handles that kind of thing, but if you know what your input JSON looks like, this solution is simple and works well.

[#9]

Thanks for your comment. Yup, agreed. Gave my answer as an alternative for @Timo Burmeister

[#10]

to me.. this is not the simple approach.. its very limiting and its no simpler than using the jolttransformation suggested above.