Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How do we get the fields of the FlowFile in nifi?

Solved Go to solution
Highlighted

How do we get the fields of the FlowFile in nifi?

 
1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: How do we get the fields of the FlowFile in nifi?

The session provides methods to read and write to the flow file content.

  1. If you are reading only then session.read with an InputStreamCallback will give you an InputStream to the flow file content
  2. If you are writing only then session.write with an OutputStreamCallback will give you an OutputStream to the flow file content
  3. If you are reading and writing at the same time then a StreamCallback will give access to the both an InputStream and OutputStream

In your case, if you are just looking to extract a value then you likely need an InputStreamCallback and you would use the InputStream to read the content and parse it appropriately for your data.

You can look at examples in the existing processors:

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-proce...

Keep in mind, the above example reads the whole content of the flow file into memory which can be dangerous when there are very large flow files, so whenever possible it is best to process the content in chunks.

View solution in original post

4 REPLIES 4
Highlighted

Re: How do we get the fields of the FlowFile in nifi?

Master Guru

@Kiran Hebbar

Hello,

Your question is not very clear as to what you are looking for. I am going to assume you are asking how to view the metadata currently associated with a FlowFile passing through your NiFi dataflow(s).

There are several ways to view this metadata:

  1. Right click on a connection that has queued data and select "List queue" from the context menu. Form the new UI that opens you will se a list of FlowFiles. Click on the 16006-screen-shot-2017-06-05-at-95354-am.png icon to the left of any one of the FlowFiles to "view details" of that FlowFile. There you will find a "attributes" tab that list all the key/value pairs associated to this FlowFile.
  2. Use data provenance to perform a search on FlowFile events. "Data Provenance" can be found under the upper right corner hamburger menu in the NiFi UI. Click the search 16007-screen-shot-2017-06-05-at-95848-am.pngicon to open a "Search Events" UI where you can add criteria to limit the results (Provenance returns 1000 of the most recent events). From the final list use the same "view details" icon to the left of an event to open a new UI that will show the Attributes of your selected FlowFile.
  3. Use the LogAttribute processor. Add this processor anywhere in your dataflow. As FlowFile pass through this processor their FlowFile Attributes as they exist at the time of passing through this processor will be logged to the nifi-app.log. Keep in mind that this processor can greatly increase the size of your logs and require more space to store your logs.

If you found this answer addressed your question, please mark it as accepted.

Thanks,

Matt

Re: How do we get the fields of the FlowFile in nifi?

@Matt Clarke

Thank you .

Let me be more specific.

How do we get the data Content from a Flowfile from inside a custom processor using the java code?

Assuming that the content of the flowfile is like this:

Name: Value

Example

Colour : Black

I need to retrieve the colour(black) from the flowfile, with java code of my custom processor.

Highlighted

Re: How do we get the fields of the FlowFile in nifi?

@Kiran Hebbar

Hi Kiran Hebbar,

I think you can use StreamCallback to read your content from a Flowfile. You can modify or add more attributes into flowfile.

Here I find a Matt Burgess's example using Javascript: http://funnifi.blogspot.com/2016/03/executescript-json-to-json-revisited.html

Hope that help you,

Thanks,

Highlighted

Re: How do we get the fields of the FlowFile in nifi?

The session provides methods to read and write to the flow file content.

  1. If you are reading only then session.read with an InputStreamCallback will give you an InputStream to the flow file content
  2. If you are writing only then session.write with an OutputStreamCallback will give you an OutputStream to the flow file content
  3. If you are reading and writing at the same time then a StreamCallback will give access to the both an InputStream and OutputStream

In your case, if you are just looking to extract a value then you likely need an InputStreamCallback and you would use the InputStream to read the content and parse it appropriately for your data.

You can look at examples in the existing processors:

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-proce...

Keep in mind, the above example reads the whole content of the flow file into memory which can be dangerous when there are very large flow files, so whenever possible it is best to process the content in chunks.

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here