Understanding FlowFile Attributes and Metadata Management in Apache NiFi

[#1]

@Rahul Soni Hi Rahul..I wanted to monitor the jvm heap usage in Nifi.So could you please suggest me a way to load flowfile attributes heavily?

[#2 · Accepted solution]

@Rahul Soni @Gillu Varghese

The GenerateFlowFile processor will create 1 GB of content for each FlowFile it creates. The FlowFile content does not live in heap memory space.
-
Each generated FlowFile will have a core set of FlowFile Attributes created. For example:

--------------------------------------------------

Standard FlowFile Attributes
Key: 'entryDate'
 Value: 'Thu Apr 26 14:52:12 UTC 2018'
Key: 'lineageStartDate'
 Value: 'Thu Apr 26 14:52:12 UTC 2018'
Key: 'fileSize'
 Value: '67'
FlowFile Attribute Map Content
Key: 'filename'
 Value: '7409235136254821'
Key: 'path'
 Value: './'
Key: 'uuid'
 Value: '119b16a1-7cb2-40ff-b92e-77bc733389e6'
--------------------------------------------------

-

You can however define attributes on attributes on each generated FlowFile by adding attributes via custom properties in GenerateFlowFile processor:

FileSize in the case of heap does not matter when it comes to heap usage of queued FlowFiles.

In this way you can create FlowFiles with as many attributes as you want:

--------------------------------------------------
Standard FlowFile Attributes
Key: 'entryDate'
 Value: 'Thu Apr 26 14:55:54 UTC 2018'
Key: 'lineageStartDate'
 Value: 'Thu Apr 26 14:55:54 UTC 2018'
Key: 'fileSize'
 Value: '0'
FlowFile Attribute Map Content
Key: 'attr1'
 Value: 'This is a test'
Key: 'attr2'
 Value: 'This is a test'
Key: 'attr3'
 Value: 'This is a test'
Key: 'attr4'
 Value: 'This is a test'
Key: 'filename'
 Value: '7409457340083769'
Key: 'path'
 Value: './'
Key: 'uuid'
 Value: 'f6254149-be47-46f3-a659-c5126ae80481'
--------------------------------------------------

-

You can adjust the run schedule and batch setting to control the number of new FlowFiles generated over a specific time period.

-

For example: Setting run Schedule to 5 sec and Batch Size to 1000, Every 5 seconds this procesor will produce 1000 new FlowFiles.

-

Thanks,

Matt

[#3]

@Matt Clarke

Thanks Matt for the information..and helping out...it worked