Both Active and Standby Namenode down in 3 node cluster on Ambari

[#1]

Hi All,

We did the disk partition activity on one of our nodes on which NN services were running. After rebooting Active NN goes down. 

Need urgent solution on this. Please help

---Version---

Ambari version -> 2.6

OS -> Linux (centos 7.9.2009)

HDP-2.6.5.1100

Attaching the required NN log

2022-04-30 06:48:30,683 WARN namenode.NameNode (NameNodeRpcServer.java:errorReport(1548)) - Fatal disk error on DatanodeRegistration(10.0.3.26:1019, datanodeUuid=a500e7d8-3052-4aa8-a18d-e768a15d22da, infoPort=1022, infoSecurePort=0, ipcPort=8010, storageInfo=lv=-56;cid=CID-dc49d075-7a70-4ddf-b66b-b39b4b445af0;nsid=897785728;c=0): DataNode failed volumes:/grid1/hadoop/hdfs/data/current;

2022-04-30 06:48:34,123 ERROR namenode.FSEditLog (JournalSet.java:mapJournalsAndReportErrors(410)) - Error: flush failed for (journal JournalAndStream(mgr=FileJournalManager(root=/grid1/hadoop/hdfs/namenode), stream=EditLogFileOutputStream(/grid1/hadoop/hdfs/namenode/current/edits_inprogress_0000000000030665138)))
java.io.IOException: Read-only file system

2022-04-30 06:48:34,124 ERROR namenode.FSEditLog (JournalSet.java:disableAndReportErrorOnJournals(360)) - Disabling journal JournalAndStream(mgr=FileJournalManager(root=/grid1/hadoop/hdfs/namenode), stream=EditLogFileOutputStream(/grid1/hadoop/hdfs/namenode/current/edits_inprogress_0000000000030665138))
2022-04-30 06:48:34,152 INFO hdfs.StateChange (FSNamesystem.java:completeFile(3759)) - DIR* completeFile: /spark-history/.e7e38c37-ff40-4961-9234-05041a654c3c is closed by DFSClient_NONMAPREDUCE_-1757037398_1

2022-04-30 06:53:54,654 ERROR namenode.NameNode (LogAdapter.java:error(71)) - RECEIVED SIGNAL 15: SIGTERM

2022-04-30 07:17:36,905 ERROR namenode.FSEditLogLoader (FSEditLogLoader.java:loadEditRecords(242)) - Encountered exception on operation MkdirOp [length=0, inodeId=11343504, path=/tmp/hive/nifi/9de63ed4-db2a-4164-b142-2e331cd008e3/hive_2022-04-06_10-25-40_878_569887095825365790-87, timestamp=1649240741042, permissions=nifi:hdfs:rwx------, aclEntries=null, opCode=OP_MKDIR, txid=29731504, xAttrs=[]]
java.lang.IllegalStateException

Any comments appreciated.

[#2 · Accepted solution]

Ensure that at OS level the Namenode disk is mounted with read/write options