I agree with @sagarshimpi , please provide the full log trace from the datanode and see if you are able to find any disk level exception in the logs are not. Also you can open NN UI-->"Datanode volume failures" tab and to confirm this. Leaving safe mode manually will not solve the problem as NN did not receive any block reports from these data nodes, if you will leave the safe mode manually then you will get missing blocks alerts for all the blocks you have (approx 34 million blocks). Did you change any configuration at hdfs level before restarting the whole HDFS service? If yes, then please share the property name. Perform a disk i/o check on all the 4 data nodes using iostat command and check if disks are working fine without any heavy %util.    ... View more

@npandey   Thanks for your response!     1. The math value for name node handler count according to Cloudera doc    dfs.namenode.service.handler.count and dfs.namenode.handler.count - For each NameNode, set to ln(number of DataNodes in this HDFS service) * 20.    https://docs.cloudera.com/documentation/enterprise/5-15-x/topics/cm_mc_autoconfig.html#concept_v4y_vb3_rn__section_bgr_d3w_d4   As per you said for example 12 data nodes it should  be   ln(12)*20 = 50   But you say to follow this 20*log2(12) formula? can you cross-check and let me know on this.   2. I have the dump but can you walk me how to check how many handlers running?     ... View more