Name Node is not starting after performing the disk expansion activity on the name node HDFS disk

Maddy2 — Mon, 02 May 2022 14:34:24 GMT

Hi All,

I have a 3 node HDP cluster and on which 2 nodes act as namenode as data node. We have recently extended the disk of HDFS from 1 tb to 3 TB. After that system got reboot and since that my name nodes are comming up.

Name node logs are mentioned below

2022-05-01 14:00:49,272 INFO namenode.FSNamesystem (FSNamesystem.java:initRetryCache(979)) - Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
2022-05-01 14:00:49,274 INFO util.GSet (LightWeightGSet.java:computeCapacity(395)) - Computing capacity for map NameNodeRetryCache
2022-05-01 14:00:49,274 INFO util.GSet (LightWeightGSet.java:computeCapacity(396)) - VM type = 64-bit
2022-05-01 14:00:49,275 INFO util.GSet (LightWeightGSet.java:computeCapacity(397)) - 0.029999999329447746% max memory 2.0 GB = 621.3 KB
2022-05-01 14:00:49,275 INFO util.GSet (LightWeightGSet.java:computeCapacity(402)) - capacity = 2^16 = 65536 entries
2022-05-01 14:00:49,299 INFO common.Storage (Storage.java:tryLock(776)) - Lock on /grid1/hadoop/hdfs/namenode/in_use.lock acquired by nodename 3716@digaudanaqamn2.gt.com
2022-05-01 14:00:49,354 INFO common.Storage (Storage.java:tryLock(776)) - Lock on /mnt/resource/hadoop/hdfs/namenode/in_use.lock acquired by nodename 3716@digaudanaqamn2.gt.com
2022-05-01 14:00:49,354 INFO namenode.FSImage (FSImage.java:recoverTransitionRead(277)) - Storage directory /mnt/resource/hadoop/hdfs/namenode is not formatted.
2022-05-01 14:00:49,354 INFO namenode.FSImage (FSImage.java:recoverTransitionRead(278)) - Formatting ...
2022-05-01 14:00:49,354 INFO common.Storage (Storage.java:clearDirectory(340)) - Will remove files: []
2022-05-01 14:00:49,355 WARN namenode.FSImage (NNStorage.java:readAndInspectDirs(1049)) - Storage directory Storage Directory /mnt/resource/hadoop/hdfs/namenode contains no VERSION file. Skipping...
2022-05-01 14:00:49,379 INFO namenode.FSImageTransactionalStorageInspector (FSImageTransactionalStorageInspector.java:inspectDirectory(85)) - No version file in /mnt/resource/hadoop/hdfs/namenode
2022-05-01 14:00:49,773 INFO namenode.FSImage (FSImage.java:loadFSImageFile(745)) - Planning to load image: FSImageFile(file=/grid1/hadoop/hdfs/namenode/current/fsimage_0000000000029731317, cpktTxId=0000000000029731317)
2022-05-01 14:00:49,868 INFO namenode.FSImageFormatPBINode (FSImageFormatPBINode.java:loadINodeSection(257)) - Loading 187768 INodes.
2022-05-01 14:00:50,935 INFO namenode.FSImageFormatProtobuf (FSImageFormatProtobuf.java:load(184)) - Loaded FSImage in 1 seconds.
2022-05-01 14:00:50,935 INFO namenode.FSImage (FSImage.java:loadFSImage(911)) - Loaded image for txid 29731317 from /grid1/hadoop/hdfs/namenode/current/fsimage_0000000000029731317
2022-05-01 14:00:50,940 INFO namenode.FSImage (FSImage.java:loadEdits(849)) - Reading org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@52eacb4b expecting start txid #29731318
2022-05-01 14:00:50,941 INFO namenode.FSImage (FSEditLogLoader.java:loadFSEdits(142)) - Start loading edits file http://digaudanaqamn2.gt.com:8480/getJournal?jid=gtqa&segmentTxId=29606282&storageInfo=-63%3A897785728%3A0%3ACID-dc49d075-7a70-4ddf-b66b-b39b4b445af0, http://digaudanaqamn3.gt.com:8480/getJournal?jid=gtqa&segmentTxId=29606282&storageInfo=-63%3A897785728%3A0%3ACID-dc49d075-7a70-4ddf-b66b-b39b4b445af0
2022-05-01 14:00:50,946 INFO namenode.RedundantEditLogInputStream (RedundantEditLogInputStream.java:nextOp(177)) - Fast-forwarding stream 'http://digaudanaqamn2.gt.com:8480/getJournal?jid=gtqa&segmentTxId=29606282&storageInfo=-63%3A897785728%3A0%3ACID-dc49d075-7a70-4ddf-b66b-b39b4b445af0, http://digaudanaqamn3.gt.com:8480/getJournal?jid=gtqa&segmentTxId=29606282&storageInfo=-63%3A897785728%3A0%3ACID-dc49d075-7a70-4ddf-b66b-b39b4b445af0' to transaction ID 29731318
2022-05-01 14:00:50,946 INFO namenode.RedundantEditLogInputStream (RedundantEditLogInputStream.java:nextOp(177)) - Fast-forwarding stream 'http://digaudanaqamn2.gt.com:8480/getJournal?jid=gtqa&segmentTxId=29606282&storageInfo=-63%3A897785728%3A0%3ACID-dc49d075-7a70-4ddf-b66b-b39b4b445af0' to transaction ID 29731318
2022-05-01 14:00:51,588 ERROR namenode.FSEditLogLoader (FSEditLogLoader.java:loadEditRecords(242)) - Encountered exception on operation MkdirOp [length=0, inodeId=11343504, path=/tmp/hive/nifi/9de63ed4-db2a-4164-b142-2e331cd008e3/hive_2022-04-06_10-25-40_878_569887095825365790-87, timestamp=1649240741042, permissions=nifi:hdfs:rwx------, aclEntries=null, opCode=OP_MKDIR, txid=29731504, xAttrs=[]]
java.lang.IllegalStateException
at com.google.common.base.Preconditions.checkState(Preconditions.java:129)
at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirForEditLog(FSDirMkdirOp.java:182)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:572)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:234)
2022-05-01 14:00:51,588 ERROR namenode.FSEditLogLoader (FSEditLogLoader.java:loadEditRecords(242)) - Encountered exception on operation MkdirOp [length=0, inodeId=11343504, path=/tmp/hive/nifi/9de63ed4-db2a-4164-b142-2e331cd008e3/hive_2022-04-06_10-25-40_878_569887095825365790-87, timestamp=1649240741042, permissions=nifi:hdfs:rwx------, aclEntries=null, opCode=OP_MKDIR, txid=29731504, xAttrs=[]]
java.lang.IllegalStateException
at com.google.common.base.Preconditions.checkState(Preconditions.java:129)
at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirForEditLog(FSDirMkdirOp.java:182)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:572)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:234)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:852)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:707)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:303)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1077)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:724)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:697)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:761)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:1001)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:985)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1710)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1778)
2022-05-01 14:00:51,695 INFO namenode.FSNamesystem (FSNamesystem.java:writeUnlock(1689)) - FSNamesystem write lock held for 2416 ms via
java.lang.Thread.getStackTrace(Thread.java:1556)
org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:945)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.writeUnlock(FSNamesystem.java:1690)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1105)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:724)
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:697)
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:761)
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:1001)
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:985)
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1710)
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1778)
Number of suppressed write-lock reports: 0
Longest write-lock held interval: 2416
2022-05-01 14:00:51,695 WARN namenode.FSNamesystem (FSNamesystem.java:loadFromDisk(726)) - Encountered exception loading fsimage
java.io.IOException: java.lang.IllegalStateException
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:244)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:852)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:707)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:303)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1077)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:724)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:707)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:303)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1077)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:724)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:697)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:761)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:1001)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:985)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1710)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1778)
Caused by: java.lang.IllegalStateException
at com.google.common.base.Preconditions.checkState(Preconditions.java:129)
at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirForEditLog(FSDirMkdirOp.java:182)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:572)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:234)
... 12 more
2022-05-01 14:00:51,719 INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@digaudanaqamn2.gt.com:50070
2022-05-01 14:00:51,820 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(211)) - Stopping NameNode metrics system...
2022-05-01 14:00:51,821 INFO impl.MetricsSinkAdapter (MetricsSinkAdapter.java:publishMetricsFromQueue(141)) - timeline thread interrupted.
2022-05-01 14:00:51,824 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(217)) - NameNode metrics system stopped.
2022-05-01 14:00:51,824 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:shutdown(606)) - NameNode metrics system shutdown complete.
2022-05-01 14:00:51,824 ERROR namenode.NameNode (NameNode.java:main(1783)) - Failed to start namenode.
java.io.IOException: java.lang.IllegalStateException
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:244)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:852)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:707)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:303)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1077)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:724)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:697)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:761)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:1001)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:985)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1710)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1778)
Caused by: java.lang.IllegalStateException
at com.google.common.base.Preconditions.checkState(Preconditions.java:129)
at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirForEditLog(FSDirMkdirOp.java:182)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:572)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:234)
... 12 more
2022-05-01 14:00:51,826 INFO util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1

Any leads ??

Re: Name Node is not starting after performing the disk expansion activity on the name node HDFS disk

Asfahan — Wed, 19 Nov 2025 09:30:20 GMT

While loading fsimage why the permissions=nifi:hdfs:rwx in-place of hdfs:hadoop
Encountered exception on operation MkdirOp

Re: Name Node is not starting after performing the disk expansion activity on the name node HDFS disk

9een — Sat, 10 Jan 2026 05:41:23 GMT

@Maddy2 FYI

➤ Based on the logs you provided, your NameNode is failing to start because it has encountered a metadata inconsistency while replaying the Edit Logs. This is a critical issue where the NameNode's current state (from the FSImage) contradicts the instructions in the Edit Logs it is trying to process.

➤ The Root Cause
The specific error is a java.lang.IllegalStateException during an OP_MKDIR operation (Transaction ID: 29731504).
The NameNode is trying to create a directory (/tmp/hive/nifi/...), but the checkState fails because the parent directory for that path does not exist in the namespace it just loaded from the FSImage.

This likely happened because:
Disk Expansion/Reboot Out of Sync: When you expanded the disk and rebooted, one of the storage directories (/mnt/resource/hadoop/hdfs/namenode) was flagged as unformatted or empty.
Metadata Corruption: There is a mismatch between your last successful checkpoint (fsimage_0000000000029731317) and the subsequent edits stored in your Journal Nodes.

➤ Recommended Solution: Metadata Recovery
Since this is an HDP (Hortonworks Data Platform) cluster with High Availability (HA), you should attempt to recover by syncing from the "good" metadata or forcing a metadata skip.

=> Step 1: Identify the Healthy NameNode
Ensure you are working on the NameNode that has the most recent and intact data. Check the other NameNode's logs to see if it also fails at the same Transaction ID.

=> Step 2:On Standby or failing Namenode kindly Check the permission of edits log and fsimage present in path dfs.namenode.name.dir
and see if it matches with permission mentioned in Active Namenode

=>Step 3: Bootstrap from the Standby (If HA is healthy)
If one NameNode is able to start or has better metadata, you can re-sync the failing node:

=> Stop the failing NameNode.
On the failing node, clear the NameNode storage directories (as defined in dfs.namenode.name.dir).
Run the bootstrap command to pull metadata from the active/healthy NameNode:

$ hdfs namenode -bootstrapStandby

4. Start the NameNode.

question Re: Name Node is not starting after performing the disk expansion activity on the name node HDFS disk in Support Questions

Name Node is not starting after performing the disk expansion activity on the name node HDFS disk

Re: Name Node is not starting after performing the disk expansion activity on the name node HDFS disk

Re: Name Node is not starting after performing the disk expansion activity on the name node HDFS disk