@Geoffrey Shelton Okot What you have mentioned is for any specific path. Do you have any procedure through which we can change the block size of existing cluster which includes all old files ? Please post it here if any you are aware!! ... View more

@ Bala Vignesh N V Thanks for your reply. Please see the data type numbers according to size. Also as you have suggested for combining files....like HAR implementation, it will be very difficult as data scattered across multiple hierarchical directory and I am not able to find any automated script/tools to do the same other than manual option. Again for manual option also I need to move data which will cause performance of the cluster. And no option for hot/cold/warm splitting as it is a single production cluster. 110 million < 1MB 19 million =>1M to 32MB 2.5 million => 32M to 64M 9 lakh => 64M to 128M 1 million > 128M Let me know your thought. ... View more

Yes I have already checked the logs as you have mentioned and nothing specific I can point out to this behavior. Have not taken thread dump but checked with diff and there seems to be some deployment of nar file but can't say specifically if that causes this. Anyway I have not faced it after that and incase if I face the same issue again I will follow your recommendations. Thanks for your help and support Matt. ... View more

Hi Matt, Sorry for the delayed response. Actually NiFi UI was not loading so that users are unable to login to NiFi. What I have done is removed the current flow.xml.gz and replaced with the latest backup of flow.xml and certainly NiFi UI was up and working. Not sure how to fix if the current flow.xml got corrupted for some reason and there were no error log which I can track for the above issue. Thanks & regards, SP ... View more

@Geoffrey Shelton OkotI Thanks for the details. I have a cluster which has 220 million files and out of which 110 million is less than 1 MB in size. Default block size is set to 128 MB. What should be the blocksize for file less than 1 MB? And How we can set in live cluster? Total Files + Directories: 227008030 Disk Remaining: 700 TB / 3.5 PB (20%) ... View more

If I create a new template, it is creating a flow.xml.gz and working fine. However If I replace the flow.xml.gz with old flow.xml.gz (one backup taken earlier in same cluster node HDF2.X) nifi UI is not coming after login and giving error as "com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out". Have tried with all parameter tuning as described by "Matt Clarke" in some other post but no results. Again If I move old file and replace with new flow.xml.gz then nifi is working fine. Please let me know if anyone faces such issues and probable reason and working around. Thanks, Suman ... View more

@Matt Clarke Have tried with all parameter tuning as described by "Matt Clarke" in some other post but no results. ... View more