Both Active and Standby Namenode down in 3 node cluster on Ambari

Kapil26 — Mon, 02 May 2022 07:56:04 GMT

Hi All,

We did the disk partition activity on one of our nodes on which NN services were running. After rebooting Active NN goes down.

Need urgent solution on this. Please help

---Version---

Ambari version -> 2.6

OS -> Linux (centos 7.9.2009)

HDP-2.6.5.1100

Attaching the required NN log

2022-04-30 06:48:30,683 WARN namenode.NameNode (NameNodeRpcServer.java:errorReport(1548)) - Fatal disk error on DatanodeRegistration(10.0.3.26:1019, datanodeUuid=a500e7d8-3052-4aa8-a18d-e768a15d22da, infoPort=1022, infoSecurePort=0, ipcPort=8010, storageInfo=lv=-56;cid=CID-dc49d075-7a70-4ddf-b66b-b39b4b445af0;nsid=897785728;c=0): DataNode failed volumes:/grid1/hadoop/hdfs/data/current;

2022-04-30 06:48:34,123 ERROR namenode.FSEditLog (JournalSet.java:mapJournalsAndReportErrors(410)) - Error: flush failed for (journal JournalAndStream(mgr=FileJournalManager(root=/grid1/hadoop/hdfs/namenode), stream=EditLogFileOutputStream(/grid1/hadoop/hdfs/namenode/current/edits_inprogress_0000000000030665138)))
java.io.IOException: Read-only file system

2022-04-30 06:48:34,124 ERROR namenode.FSEditLog (JournalSet.java:disableAndReportErrorOnJournals(360)) - Disabling journal JournalAndStream(mgr=FileJournalManager(root=/grid1/hadoop/hdfs/namenode), stream=EditLogFileOutputStream(/grid1/hadoop/hdfs/namenode/current/edits_inprogress_0000000000030665138))
2022-04-30 06:48:34,152 INFO hdfs.StateChange (FSNamesystem.java:completeFile(3759)) - DIR* completeFile: /spark-history/.e7e38c37-ff40-4961-9234-05041a654c3c is closed by DFSClient_NONMAPREDUCE_-1757037398_1

2022-04-30 06:53:54,654 ERROR namenode.NameNode (LogAdapter.java:error(71)) - RECEIVED SIGNAL 15: SIGTERM

2022-04-30 07:17:36,905 ERROR namenode.FSEditLogLoader (FSEditLogLoader.java:loadEditRecords(242)) - Encountered exception on operation MkdirOp [length=0, inodeId=11343504, path=/tmp/hive/nifi/9de63ed4-db2a-4164-b142-2e331cd008e3/hive_2022-04-06_10-25-40_878_569887095825365790-87, timestamp=1649240741042, permissions=nifi:hdfs:rwx------, aclEntries=null, opCode=OP_MKDIR, txid=29731504, xAttrs=[]]
java.lang.IllegalStateException

Any comments appreciated.

Re: Both Active and Standby Namenode down in 3 node cluster on Ambari

rki_ — Wed, 04 May 2022 07:26:31 GMT

Ensure that at OS level the Namenode disk is mounted with read/write options

question Both Active and Standby Namenode down in 3 node cluster on Ambari in Support Questions

Both Active and Standby Namenode down in 3 node cluster on Ambari

Re: Both Active and Standby Namenode down in 3 node cluster on Ambari