Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi - How to add key:value to json

Highlighted

Nifi - How to add key:value to json

New Contributor

Hi, we need add a default key:value with actual timestamp to the following json input structure:

[ {
  "host" : "DUSTSADMIN.ads.xyz.de",
  "host_icons" : "menu",
  "host_state" : "UP",
  "num_services_crit" : "0",
  "num_services_ok" : "28",
  "num_services_pending" : "0",
  "num_services_unknown" : "0",
  "num_services_warn" : "0"
}, {
  "host" : "DUSTSVMDC01.ads.xyz.de",
  "host_icons" : "menu",
  "host_state" : "UP",
  "num_services_crit" : "0",
  "num_services_ok" : "34",
  "num_services_pending" : "0",
  "num_services_unknown" : "0",
  "num_services_warn" : "0"
}, {
  "host" : "DUSTSVMDC02.ads.xyz.de",
  "host_icons" : "menu",
  "host_state" : "UP",
  "num_services_crit" : "0",
  "num_services_ok" : "34",
  "num_services_pending" : "0",
  "num_services_unknown" : "0",
  "num_services_warn" : "0"
} ]

needed output:

[ { "ctime" : "Current_timestamp",
  "host" : "DUSTSADMIN.ads.xyz.de",
  "host_icons" : "menu",
  "host_state" : "UP",
  "num_services_crit" : "0",
  "num_services_ok" : "28",
  "num_services_pending" : "0",
  "num_services_unknown" : "0",
  "num_services_warn" : "0"
}, {"ctime" : "Current_timestamp",
  "host" : "DUSTSVMDC01.ads.xyz.de",
  "host_icons" : "menu",
  "host_state" : "UP",
  "num_services_crit" : "0",
  "num_services_ok" : "34",
  "num_services_pending" : "0",
  "num_services_unknown" : "0",
  "num_services_warn" : "0"
}, {"ctime" : "Current_timestamp",
  "host" : "DUSTSVMDC02.ads.xyz.de",
  "host_icons" : "menu",
  "host_state" : "UP",
  "num_services_crit" : "0",
  "num_services_ok" : "34",
  "num_services_pending" : "0",
  "num_services_unknown" : "0",
  "num_services_warn" : "0"
} ]

9 REPLIES 9

Re: Nifi - How to add key:value to json

Expert Contributor

One way to do this is to first use the EvaluateJsonPath processor and break out the json into individual attributes. Notice that you can add ctime by clicking the + symbol, adding a name and a value.

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.2.0/org.apache...

See image, evaluatejsonpath, below.

Then later, you can use the AttributesToJson processor to rebuild the complete JSON event.

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.2.0/org.apache...

See image, attributestojson, below.The output at this point would look like this

{
  "twitterMessage" : "20x faster VM access w/ Dell EMC and Intel. Learn more  https://t.co/1hD0MYfBbh https://t.co/0GhsMQve8b",
  "twitterUser" : "GCSIT Solutions",
  "ctime" : "Wed Jun 07 18:10:35 +0000 2017"
}

attributestojson.pngevaluatejsonpath.png

Re: Nifi - How to add key:value to json

In addition to @bhagan's answer, as of NiFi 1.2.0 (due to NIFI-3010), you can use NiFi Expression Language in JOLT specifications, so with a JoltTransformJSON processor you could have the following Chain spec:

[{
  "operation": "default",
  "spec": {
    "*": {
      "ctime": "${now()}"
    }
  }
}]

If you want the ctime field to contain a string with the number of milliseconds since Epoch, you can use now():toNumber() instead of just now(). If you want an actual numeric value (rather than a String containing the numeric value), you need an additional operation to change the value to a long:

[{
  "operation": "default",
  "spec": {
    "*": {
      "ctime": "${now():toNumber()}"
    }
  }
}, {
  "operation": "modify-overwrite-beta",
  "spec": {
    "*": {
      "ctime": "=toLong"
    }
  }
}]

Re: Nifi - How to add key:value to json

Expert Contributor

@Matt Burgess, Love JOLT transformation solution, we use it a lot with dynamic jsons, transpose, etc... but in this case, I would go simple, just replace text. let me know if I miss something (check my alternative)

Re: Nifi - How to add key:value to json

New Contributor

Hi @Matt Burgess.

First I want do say thank you for your work!

Second, could you please update your answer. The first part about adding an element works like charm, but when I add the part about modifying string to long the JoltTransformJSON processor thorws an error.

\" " is invalid because Specification not valid for the selected transformation \"

I'm using Nifi 1.4.0

Re: Nifi - How to add key:value to json

New Contributor

this worked for me so easily.. i tried the replacetext suggested below.. but it replaced every {. Jolt seems like the clean and easy way to do this. Kudos @Matt Burgess

Re: Nifi - How to add key:value to json

Expert Contributor

@Timo Burmeister,

Keep it simple:

use replaceText processor with this configuration:

Search Value         :  [{]
Replacement Value    :  { "ctime":"${now()}",
Replacement Strategy :  Regex Replace
Evaluation Mode      :  Entire text

Re: Nifi - How to add key:value to json

True, if you are just inserting a single new field at the top-level of a flat JSON doc, this should work fine. However you may want an optional newline between the array and the first object if the JSON is pretty printed. Also, would this work if you had a nested array of objects further down in the file? I guess you could put in a start-of-line into the regex, but if the JSON has weird whitespacing you may run into the same problem. I recommend JOLT in general because it handles that kind of thing, but if you know what your input JSON looks like, this solution is simple and works well.

Re: Nifi - How to add key:value to json

Expert Contributor

Thanks for your comment. Yup, agreed. Gave my answer as an alternative for @Timo Burmeister

Re: Nifi - How to add key:value to json

New Contributor

to me.. this is not the simple approach.. its very limiting and its no simpler than using the jolttransformation suggested above.

Don't have an account?
Coming from Hortonworks? Activate your account here