flowfile attributes in Nifi

gillu_rock_in — Thu, 26 Apr 2018 17:57:44 GMT

@Rahul Soni Hi Rahul..I wanted to monitor the jvm heap usage in Nifi.So could you please suggest me a way to load flowfile attributes heavily?

Re: flowfile attributes in Nifi

MattWho — Sun, 18 Aug 2019 09:09:35 GMT

@Rahul Soni @Gillu Varghese

The GenerateFlowFile processor will create 1 GB of content for each FlowFile it creates. The FlowFile content does not live in heap memory space.
-
Each generated FlowFile will have a core set of FlowFile Attributes created. For example:

--------------------------------------------------

Standard FlowFile Attributes
Key: 'entryDate'
 Value: 'Thu Apr 26 14:52:12 UTC 2018'
Key: 'lineageStartDate'
 Value: 'Thu Apr 26 14:52:12 UTC 2018'
Key: 'fileSize'
 Value: '67'
FlowFile Attribute Map Content
Key: 'filename'
 Value: '7409235136254821'
Key: 'path'
 Value: './'
Key: 'uuid'
 Value: '119b16a1-7cb2-40ff-b92e-77bc733389e6'
--------------------------------------------------

You can however define attributes on attributes on each generated FlowFile by adding attributes via custom properties in GenerateFlowFile processor:

FileSize in the case of heap does not matter when it comes to heap usage of queued FlowFiles.

In this way you can create FlowFiles with as many attributes as you want:

--------------------------------------------------
Standard FlowFile Attributes
Key: 'entryDate'
 Value: 'Thu Apr 26 14:55:54 UTC 2018'
Key: 'lineageStartDate'
 Value: 'Thu Apr 26 14:55:54 UTC 2018'
Key: 'fileSize'
 Value: '0'
FlowFile Attribute Map Content
Key: 'attr1'
 Value: 'This is a test'
Key: 'attr2'
 Value: 'This is a test'
Key: 'attr3'
 Value: 'This is a test'
Key: 'attr4'
 Value: 'This is a test'
Key: 'filename'
 Value: '7409457340083769'
Key: 'path'
 Value: './'
Key: 'uuid'
 Value: 'f6254149-be47-46f3-a659-c5126ae80481'
--------------------------------------------------

You can adjust the run schedule and batch setting to control the number of new FlowFiles generated over a specific time period.

For example: Setting run Schedule to 5 sec and Batch Size to 1000, Every 5 seconds this procesor will produce 1000 new FlowFiles.

Thanks,

Matt

Re: flowfile attributes in Nifi

gillu_rock_in — Fri, 27 Apr 2018 13:11:19 GMT

@Matt Clarke

Thanks Matt for the information..and helping out...it worked

question Re: flowfile attributes in Nifi in Archives of Support Questions (Read Only)

flowfile attributes in Nifi

Re: flowfile attributes in Nifi

Re: flowfile attributes in Nifi