Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi - Script for JSON-content manipulation before JOLT

NiFi - Script for JSON-content manipulation before JOLT

Rising Star

Hi all,

as far as I investigated this (https://community.hortonworks.com/questions/247330/nifi-need-help-with-jolt-syntax-jolttransformjson-1.html) it is not possible to do this with JOLT.

Now I want to manipulate the JSON-content BEFORE JOLT does simple transformation.

I'm getting used to do some things with groovy, viewed some examples but couldn't find the right syntax.


There are two things I need to to:


1. All timestamps from JSON have to be converted and formated from CET yyyyMMddHHmmss ==> to UTC yyyy-MM-dd HH:mm:ss.SSS


2. Some string manipulation with some kind of expression language doing this:

Expression Language to build "dnr_group"


   ${DNR:substringAfter('('):substringBefore(')'):toNumber()
                            :lt( 10 )
                                     :ifElse(${DNR:substringAfter('('):substringBefore(')'):prepend('0')},
                                             ${DNR:substringAfter('('):substringBefore(')')})
                            :append('-'):append(${GROUP:toNumber()
                            :lt( 10 )
                                     :ifElse(${GROUP:prepend('0')},
                                             ${GROUP})})
    :prepend('V-')}

Input-JSON

{
  "user": "Justen",
  "stand": "20190702121621",
  "table": [
    {
      "zn": 1,
      "elem_user": "Meier",
      "elem_stand": "20190705081410",
      "dnr": "(0)",
      "group": "1"
    },
    {
      "elem_stand": "20190706201918",
      "zn": 2,
      "elem_user": "Schmidt",
      "dnr": "(0)",
      "group": "2"
    }
  ]
}

Desired Output-JSON

    {
        "user": "Justen",
        "stand": "2019-07-02 10:16:21",
        "table": [
            {
                "zn": 1,
                "elem_user": "Meier",
                "elem_stand": "2019-07-05 06:14:10",
                "dnr_group": "V-00-01"
            },
            {
                "elem_stand": "2019-07-06 18:19:18",
                "zn": 2,
                "elem_user": "Schmidt",
                "dnr_group": "V-00-02"
            }
        ]
    }

Is it possible to do this with groovy or does anybody know other ways manipulating JSON-content in NiFi?

Thanks for any help!


2 REPLIES 2

Re: NiFi - Script for JSON-content manipulation before JOLT

New Contributor

you could first read that line from the json file (evaluate json path) and then update attribute with expression language and push it back into the json file with update record processor.


let me know if you need further help.

Highlighted

Re: NiFi - Script for JSON-content manipulation before JOLT

Rising Star

Hi @Rosa Negra thanks for your answer!

I worked with ff-attributes as long as the JSON-file contained only one entry at a time and it worked fine.

Now with the complex structure I couldn't find a way to manipulate the fields of the positions.


Unfortunately I have no experience with record processing. If I understand you correctly it would be possible to update the JSON doing record processing.

So I am "afraid" I have to explore it...


Maybe there are further solutions?

Don't have an account?
Coming from Hortonworks? Activate your account here