Hi All,
We did the disk partition activity on one of our nodes on which NN services were running. After rebooting Active NN goes down.
Need urgent solution on this. Please help
---Version---
Ambari version -> 2.6
OS -> Linux (centos 7.9.2009)
HDP-2.6.5.1100
Attaching the required NN log
2022-04-30 06:48:30,683 WARN namenode.NameNode (NameNodeRpcServer.java:errorReport(1548)) - Fatal disk error on DatanodeRegistration(10.0.3.26:1019, datanodeUuid=a500e7d8-3052-4aa8-a18d-e768a15d22da, infoPort=1022, infoSecurePort=0, ipcPort=8010, storageInfo=lv=-56;cid=CID-dc49d075-7a70-4ddf-b66b-b39b4b445af0;nsid=897785728;c=0): DataNode failed volumes:/grid1/hadoop/hdfs/data/current;
2022-04-30 06:48:34,123 ERROR namenode.FSEditLog (JournalSet.java:mapJournalsAndReportErrors(410)) - Error: flush failed for (journal JournalAndStream(mgr=FileJournalManager(root=/grid1/hadoop/hdfs/namenode), stream=EditLogFileOutputStream(/grid1/hadoop/hdfs/namenode/current/edits_inprogress_0000000000030665138)))
java.io.IOException: Read-only file system
2022-04-30 06:48:34,124 ERROR namenode.FSEditLog (JournalSet.java:disableAndReportErrorOnJournals(360)) - Disabling journal JournalAndStream(mgr=FileJournalManager(root=/grid1/hadoop/hdfs/namenode), stream=EditLogFileOutputStream(/grid1/hadoop/hdfs/namenode/current/edits_inprogress_0000000000030665138))
2022-04-30 06:48:34,152 INFO hdfs.StateChange (FSNamesystem.java:completeFile(3759)) - DIR* completeFile: /spark-history/.e7e38c37-ff40-4961-9234-05041a654c3c is closed by DFSClient_NONMAPREDUCE_-1757037398_1
2022-04-30 06:53:54,654 ERROR namenode.NameNode (LogAdapter.java:error(71)) - RECEIVED SIGNAL 15: SIGTERM
2022-04-30 07:17:36,905 ERROR namenode.FSEditLogLoader (FSEditLogLoader.java:loadEditRecords(242)) - Encountered exception on operation MkdirOp [length=0, inodeId=11343504, path=/tmp/hive/nifi/9de63ed4-db2a-4164-b142-2e331cd008e3/hive_2022-04-06_10-25-40_878_569887095825365790-87, timestamp=1649240741042, permissions=nifi:hdfs:rwx------, aclEntries=null, opCode=OP_MKDIR, txid=29731504, xAttrs=[]]
java.lang.IllegalStateException
Any comments appreciated.
