Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi : combine flow file attribute and content

Solved Go to solution
Highlighted

NiFi : combine flow file attribute and content

Explorer

Hello,

I am puzzelling for some time if there is possibility to refer the both attributes and content of flow file and transform the content by combining both.
Example:
Assume a flow file with uuid=2c8a7845-8567-4eed-97ee-4dcd1c18668e and the json content

{
   "data":"some business data"
}

 

The result of the transformation should be a content like

{
   "nifi-filename": "2c8a7845-8567-4eed-97ee-4dcd1c18668e",
   "nifi-content": {
       "data": "some business data"
   }
}

 

How can I reach that.?

 

Best regards
Jaro

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: NiFi : combine flow file attribute and content

Explorer

Hello,

 

You can create a nested node "nifi-content" using the ReplaceText processor.

You just need to configure the Replacement Value field :

 

{"nifi-content":$1}

 

 

Then just add a new node "filename" using the JoltTransformJSON processor with following Jolt specification :

 

[
  {
    "operation": "default",
    "spec": {
      "filename": "${filename}"
    }
}, {
    "operation": "default",
    "spec": {
      "*": "&"
    }
}
]

 

 

Sans titre.png

 

Maybe you can create the "nifi-content" node and add the "filename" node only using one JoltTransform JSON ; but i'm not very experienced using Jolt and I did not find out how.

View solution in original post

4 REPLIES 4
Highlighted

Re: NiFi : combine flow file attribute and content

@Jarinek   Yes, this is totally possible in NiFi.   Traditionally one method to do this could be something like EvaluateJsonPath to get some payload json values to attributes and then use AttributesToJson to reform a new json object of all the attributes you have created.   In newer versions of Nifi you have some other options with QueryRecord, UpdateRecord and the Json Record Readers, Record Writers.  I think another option is also custom ExecuteScript and JoltTransform on JSON.

 

 

My suggestion would be to research NiFi + JSON and begin with some simple examples.  Once you have some experience with basic examples, begin to transform them to suit your use case.   I also suggest that you try it in different ways before deciding on one.  For example,  you may build a flow and get it working (evaluateJsonPath), but improve it over time based on new nifi versions and its capabilities (Use JSON Record Reader/Writers).

 

 

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  

 

Thanks,


Steven @ DFHZ

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven

Highlighted

Re: NiFi : combine flow file attribute and content

Explorer

Well, EvaluateJsonPath sounds promising but it looks like that EvaluateJsonPath is somehow limitated.
It is unable to extract json objects, only single attributes, which makes it impossible to use for slightly more complex data.
Let's suppose this.

Input:
{
   whatever known or unknown json nested structures here
}

Desired output
{
    "nifi-filename": "2c8a7845-8567-4eed-97ee-4dcd1c18668e",
    "nifi-content": {
       whatever known or unknown json possibly structures or sub-structures of the input message here
    }
}

Highlighted

Re: NiFi : combine flow file attribute and content

Explorer

Hello,

 

You can create a nested node "nifi-content" using the ReplaceText processor.

You just need to configure the Replacement Value field :

 

{"nifi-content":$1}

 

 

Then just add a new node "filename" using the JoltTransformJSON processor with following Jolt specification :

 

[
  {
    "operation": "default",
    "spec": {
      "filename": "${filename}"
    }
}, {
    "operation": "default",
    "spec": {
      "*": "&"
    }
}
]

 

 

Sans titre.png

 

Maybe you can create the "nifi-content" node and add the "filename" node only using one JoltTransform JSON ; but i'm not very experienced using Jolt and I did not find out how.

View solution in original post

Re: NiFi : combine flow file attribute and content

Explorer

Actually, both replies can be considered as valid. I confirmed that one, which better fits to my use case.

Don't have an account?
Coming from Hortonworks? Activate your account here