Support Questions
Find answers, ask questions, and share your expertise

Both Active and Standby Namenode down in 3 node cluster on Ambari

New Contributor

Hi All,

 

We did the disk partition activity on one of our nodes on which NN services were running. After rebooting Active NN goes down. 

Need urgent solution on this. Please help

 

---Version---

Ambari version -> 2.6

OS -> Linux (centos 7.9.2009)

HDP-2.6.5.1100

Attaching the required NN log

 

2022-04-30 06:48:30,683 WARN namenode.NameNode (NameNodeRpcServer.java:errorReport(1548)) - Fatal disk error on DatanodeRegistration(10.0.3.26:1019, datanodeUuid=a500e7d8-3052-4aa8-a18d-e768a15d22da, infoPort=1022, infoSecurePort=0, ipcPort=8010, storageInfo=lv=-56;cid=CID-dc49d075-7a70-4ddf-b66b-b39b4b445af0;nsid=897785728;c=0): DataNode failed volumes:/grid1/hadoop/hdfs/data/current;

 

2022-04-30 06:48:34,123 ERROR namenode.FSEditLog (JournalSet.java:mapJournalsAndReportErrors(410)) - Error: flush failed for (journal JournalAndStream(mgr=FileJournalManager(root=/grid1/hadoop/hdfs/namenode), stream=EditLogFileOutputStream(/grid1/hadoop/hdfs/namenode/current/edits_inprogress_0000000000030665138)))
java.io.IOException: Read-only file system

 

2022-04-30 06:48:34,124 ERROR namenode.FSEditLog (JournalSet.java:disableAndReportErrorOnJournals(360)) - Disabling journal JournalAndStream(mgr=FileJournalManager(root=/grid1/hadoop/hdfs/namenode), stream=EditLogFileOutputStream(/grid1/hadoop/hdfs/namenode/current/edits_inprogress_0000000000030665138))
2022-04-30 06:48:34,152 INFO hdfs.StateChange (FSNamesystem.java:completeFile(3759)) - DIR* completeFile: /spark-history/.e7e38c37-ff40-4961-9234-05041a654c3c is closed by DFSClient_NONMAPREDUCE_-1757037398_1

 

2022-04-30 06:53:54,654 ERROR namenode.NameNode (LogAdapter.java:error(71)) - RECEIVED SIGNAL 15: SIGTERM

2022-04-30 07:17:36,905 ERROR namenode.FSEditLogLoader (FSEditLogLoader.java:loadEditRecords(242)) - Encountered exception on operation MkdirOp [length=0, inodeId=11343504, path=/tmp/hive/nifi/9de63ed4-db2a-4164-b142-2e331cd008e3/hive_2022-04-06_10-25-40_878_569887095825365790-87, timestamp=1649240741042, permissions=nifi:hdfs:rwx------, aclEntries=null, opCode=OP_MKDIR, txid=29731504, xAttrs=[]]
java.lang.IllegalStateException

 

Any comments appreciated.

 

 

1 REPLY 1

Cloudera Employee

Ensure that at OS level the Namenode disk is mounted with read/write options

; ;