Agreed, but is there a way to avoid this wastage. apart from migrating data to LFS and then again to HDFS. Example: We have a 500MB file with block size 128 MB i.e. 4 blocks on HDFS. Now since we changed block size to 256MB, how would we make the file on HDFS to have 2 blocks of 256MB instead of 4. Please suggest.  ... View more

@ Bala Vignesh N V Thanks for your reply. Please see the data type numbers according to size. Also as you have suggested for combining files....like HAR implementation, it will be very difficult as data scattered across multiple hierarchical directory and I am not able to find any automated script/tools to do the same other than manual option. Again for manual option also I need to move data which will cause performance of the cluster. And no option for hot/cold/warm splitting as it is a single production cluster. 110 million < 1MB 19 million =>1M to 32MB 2.5 million => 32M to 64M 9 lakh => 64M to 128M 1 million > 128M Let me know your thought. ... View more

Yes I have already checked the logs as you have mentioned and nothing specific I can point out to this behavior. Have not taken thread dump but checked with diff and there seems to be some deployment of nar file but can't say specifically if that causes this. Anyway I have not faced it after that and incase if I face the same issue again I will follow your recommendations. Thanks for your help and support Matt. ... View more

@Geoffrey Shelton Okot What you have mentioned is for any specific path. Do you have any procedure through which we can change the block size of existing cluster which includes all old files ? Please post it here if any you are aware!! ... View more

If I create a new template, it is creating a flow.xml.gz and working fine. However If I replace the flow.xml.gz with old flow.xml.gz (one backup taken earlier in same cluster node HDF2.X) nifi UI is not coming after login and giving error as "com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out". Have tried with all parameter tuning as described by "Matt Clarke" in some other post but no results. Again If I move old file and replace with new flow.xml.gz then nifi is working fine. Please let me know if anyone faces such issues and probable reason and working around. Thanks, Suman ... View more