Both Active and Standby Namenode down in 3 node cluster on Ambari
Hi All,
We did the disk partition activity on one of our nodes on which NN services were running. After rebooting Active NN goes down.
Need urgent solution on this. Please help
---Version---
Ambari version -> 2.6
OS -> Linux (centos 7.9.2009)
HDP-2.6.5.1100
Attaching the required NN log
2022-04-30 06:48:30,683 WARN namenode.NameNode (NameNodeRpcServer.java:errorReport(1548)) - Fatal disk error on DatanodeRegistration(10.0.3.26:1019, datanodeUuid=a500e7d8-3052-4aa8-a18d-e768a15d22da, infoPort=1022, infoSecurePort=0, ipcPort=8010, storageInfo=lv=-56;cid=CID-dc49d075-7a70-4ddf-b66b-b39b4b445af0;nsid=897785728;c=0): DataNode failed volumes:/grid1/hadoop/hdfs/data/current;
2022-04-30 06:48:34,123 ERROR namenode.FSEditLog (JournalSet.java:mapJournalsAndReportErrors(410)) - Error: flush failed for (journal JournalAndStream(mgr=FileJournalManager(root=/grid1/hadoop/hdfs/namenode), stream=EditLogFileOutputStream(/grid1/hadoop/hdfs/namenode/current/edits_inprogress_0000000000030665138)))
java.io.IOException: Read-only file system
2022-04-30 06:48:34,124 ERROR namenode.FSEditLog (JournalSet.java:disableAndReportErrorOnJournals(360)) - Disabling journal JournalAndStream(mgr=FileJournalManager(root=/grid1/hadoop/hdfs/namenode), stream=EditLogFileOutputStream(/grid1/hadoop/hdfs/namenode/current/edits_inprogress_0000000000030665138))
2022-04-30 06:48:34,152 INFO hdfs.StateChange (FSNamesystem.java:completeFile(3759)) - DIR* completeFile: /spark-history/.e7e38c37-ff40-4961-9234-05041a654c3c is closed by DFSClient_NONMAPREDUCE_-1757037398_1
2022-04-30 06:53:54,654 ERROR namenode.NameNode (LogAdapter.java:error(71)) - RECEIVED SIGNAL 15: SIGTERM
2022-04-30 07:17:36,905 ERROR namenode.FSEditLogLoader (FSEditLogLoader.java:loadEditRecords(242)) - Encountered exception on operation MkdirOp [length=0, inodeId=11343504, path=/tmp/hive/nifi/9de63ed4-db2a-4164-b142-2e331cd008e3/hive_2022-04-06_10-25-40_878_569887095825365790-87, timestamp=1649240741042, permissions=nifi:hdfs:rwx------, aclEntries=null, opCode=OP_MKDIR, txid=29731504, xAttrs=[]]
java.lang.IllegalStateException
Any comments appreciated.
