Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Is it the best practice to delete the user defined attributes after they have been used at the end of the NiFi flow?

Solved Go to solution
Highlighted

Is it the best practice to delete the user defined attributes after they have been used at the end of the NiFi flow?

New Contributor

Hi, 

As  I learnt that the flow file attributes are held in memory in the nifi process, is it a good idea to clean up the attributes that have been defined earlier and delete them using UpdateAttribute processor if we know that we wouldn't use them anymore ? Or does deleting doesn't matter at all with regards to the memory usage? Please clarify.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Is it the best practice to delete the user defined attributes after they have been used at the end of the NiFi flow?

Yes.  Auto-termination is how you drop the whole flowfile and all attributes.    

 

If for example, you have a giant attribute of SCHEMA or JSON,  using updateAttribute could be used to empty the value.   However if you do not even need the flowfile anymore, auto-terminate, versus updateAttribute and retain flowfile.

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven

View solution in original post

4 REPLIES 4
Highlighted

Re: Is it the best practice to delete the user defined attributes after they have been used at the end of the NiFi flow?

@Jayavardhini Yes, It is a best practice to auto-terminate successfully completed data flow flowfiles at the bottom of your flow branches.  You do not want them to remain as they will continue to hold resources.  

 

A good NiFi developer will build a flow during development where all bottom branches are visible, including routing all processor relationships, even ones that will eventually be auto terminated.   This gives visibility during testing and flow creation if someone unexpected happens.  I use stopped output ports for this purpose.  

 

Screen Shot 2020-05-28 at 8.42.10 AM.png

 

In some of my production flows I create capture points for exceptions, these are bottom branch Process Groups or Queues where the flowfiles will remain until someone inspects them, makes a change, inspects the provenance, and maybe even reroutes it back into the flow again.   This is the only case where I keep flowfiles in my flow.  In all other cases I auto terminate and the flowfiles are gone from my flows.

 

 

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  

 

Thanks,


Steven @ DFHZ

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven

Highlighted

Re: Is it the best practice to delete the user defined attributes after they have been used at the end of the NiFi flow?

New Contributor

Thanks Steven. But my question was specifically to the attributes that we define when needed in the flow and if the operation of deleting them at the end is particularly useful or not to improve the memory usage ?  So, i am just expecting an answer about the attributes here.

Highlighted

Re: Is it the best practice to delete the user defined attributes after they have been used at the end of the NiFi flow?

Yes.  Auto-termination is how you drop the whole flowfile and all attributes.    

 

If for example, you have a giant attribute of SCHEMA or JSON,  using updateAttribute could be used to empty the value.   However if you do not even need the flowfile anymore, auto-terminate, versus updateAttribute and retain flowfile.

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven

View solution in original post

Highlighted

Re: Is it the best practice to delete the user defined attributes after they have been used at the end of the NiFi flow?

New Contributor

Oh, that's great. Thanks for your response. That clarifies my question. 

Don't have an account?
Coming from Hortonworks? Activate your account here