HDFS to many bad blocks due to" Operation category WRITE is not supported in state standby" - Understanding why datanode can't find Active NaneNode

[#1]

Recently I want to upgrade our cluster from 2.6.5 to 3.1.3 but failed.so I rollback the version to old.but some strange things hanppend .our cluster's datanode can't report the block's situation to the the Active NameNode. so the datanode throw this exception "

"

I don't know why. The datanode throw this problem all the time. And the NameNode Web UI show the "There are xxx missing blocks. The following files may be corrupted" the information but the number of missing blocks still rising。。。。really scary 

I don't know what happend to our cluster.. ....

[#2]

@datafiber Welcome to our community! To help you get the best possible answer, I have tagged in our HDFS experts @SVB @Asok @rki_ who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

[#3]

Hi @datafiber it seems like your Namenode is in Safe mode, not sure why it went into safe mode, but you can try taking it out manually and then retry the operation and monitor the logs.

run the below commands from NN.

# hdfs dfsadmin -safemode leave
# hdfs dfsadmin -safemode status