@Maddy2 FYI ➤ Based on the logs you provided, your NameNode is failing to start because it has encountered a metadata inconsistency while replaying the Edit Logs. This is a critical issue where the NameNode's current state (from the FSImage) contradicts the instructions in the Edit Logs it is trying to process. ➤ The Root Cause The specific error is a java.lang.IllegalStateException during an OP_MKDIR operation (Transaction ID: 29731504). The NameNode is trying to create a directory (/tmp/hive/nifi/...), but the checkState fails because the parent directory for that path does not exist in the namespace it just loaded from the FSImage. This likely happened because: Disk Expansion/Reboot Out of Sync: When you expanded the disk and rebooted, one of the storage directories (/mnt/resource/hadoop/hdfs/namenode) was flagged as unformatted or empty. Metadata Corruption: There is a mismatch between your last successful checkpoint (fsimage_0000000000029731317) and the subsequent edits stored in your Journal Nodes. ➤ Recommended Solution: Metadata Recovery Since this is an HDP (Hortonworks Data Platform) cluster with High Availability (HA), you should attempt to recover by syncing from the "good" metadata or forcing a metadata skip. => Step 1: Identify the Healthy NameNode Ensure you are working on the NameNode that has the most recent and intact data. Check the other NameNode's logs to see if it also fails at the same Transaction ID. => Step 2:On Standby or failing Namenode kindly Check the permission of edits log and fsimage present in path dfs.namenode.name.dir and see if it matches with permission mentioned in Active Namenode =>Step 3: Bootstrap from the Standby (If HA is healthy) If one NameNode is able to start or has better metadata, you can re-sync the failing node: => Stop the failing NameNode. On the failing node, clear the NameNode storage directories (as defined in dfs.namenode.name.dir). Run the bootstrap command to pull metadata from the active/healthy NameNode: $ hdfs namenode -bootstrapStandby 4. Start the NameNode. ... View more

1. Does cp just restore the inode? No. When you use the cp command, the system does not simply "re-link" the old metadata or inode. Instead, it creates brand-new files. The cp command reads the data blocks from the snapshot and writes them to the target directory as new blocks. The resulting files will have new Inode IDs, new timestamps, and your current user as the owner. It is a heavy operation because it physically duplicates the data on the disk (until the background deduplication/hard-linking handles it, depending on the underlying filesystem). 2. What happens to unmodified files? By default, the hdfs dfs -cp command will overwrite the existing files in the target directory even if they haven't changed. It does not perform a "diff" or a check to see if the content is identical. It will read the file from the snapshot and overwrite the live file, resulting in unnecessary I/O and network traffic. 3. Are those files skipped? No. Using cp with a wildcard (*) will force the system to attempt to copy everything. If a file with the same name exists in the target, the command will fail with a "File exists" error unless you use specific flags (like -f in some environments) or delete the target first. ➤  The "Better" Way (Recommended Fix) If the goal is to only restore metadata or only update files that actually changed, using cp is inefficient. Instead, suggest the following: A. Use distcp with the -update flag distcp is much smarter. It compares the source (snapshot) and the target (live) and only copies files that have different sizes or checksums. hadoop distcp -update -ptag <snaproot>/.snapshot/<name>/ <target_dir>/ -update: Only copies files if the size/checksum differs. -ptag: Preserves the original permissions, timestamps, and ACLs. B. Manual "Restore" (Metadata only) If you only changed permissions (chmod) and didn't touch the data, the most efficient "restore" isn't a copy at all—it’s simply running chmod again to set them back. Snapshots are great for insurance, but they are most useful for data recovery, not metadata undoing. ... View more

✥ In CDH6 HBase, the property was removed per https://issues.apache.org/jira/browse/HBASE-15989 because we allow all altering operations without disabling table. cc : @webtube  ... View more

✥ For the Error "Unrecognized option:-j" kindly manually type the keyword -j instead of copy-paste => Note: you generally notice such exceptions when you have special character pasted ✥ Kindly make sure you first bypass the stuck procedure and locks $ hbase hbck -j /tmp/target/hbase-hbck2-1.3.0-SNAPSHOT.jar bypass -o -r <pid> => For reviewing the stuck procedure kindly navigate to Hmaster ui => Procedure & locks and first bypass the proc id visible in lock section ✥ The you can consider closing the region State before disabling table $ hbase hbck -j /tmp/target/hbase-hbck2-1.3.0-SNAPSHOT.jar setRegionState $i CLOSED $ hbase hbck -j /tmp/target/hbase-hbck2-1.3.0-SNAPSHOT.jar setTableState <tablename> DISABLED => Note: Make sure the 3k regions are all part of the same table " " which you wish to disable/remove ✥ Once table is disabled then you can login into hbase shell and then perform drop table operation $ drop 'tablename' ... View more

edits file: An edits file contains a log of all transactions after the most recent fsimage file and  contains the transactions of the file system changes (like create file, delete file, permissions change etc) The Checkpointing process will then periodically merge the content of the most recent fsimage with the edits (containing new transactions) to create a new fsimage. Although the edits log file are redundant after they are merged in fsimage, they are kept for safety/potential recovery requirement reasons and is part of the regular design. This should be finite by default however. The two configuration parameters to control this are: a. "dfs.namenode.num.extra.edits.retained" (default 1000000) : This determines many transactions to keep, regardless of how many edit files they are spread across. b. "dfs.namenode.max.extra.edits.segments.retained" (default 10000). This serves as a secondary cap for the former. This means that around 10000 extra files would be kept at all times, as long as those 10000 files keep about 1 million edits (per above configuration parameter). On a healthy, periodic checkpointing cluster, each edit file should not be higher than ~2-5MB, and thus the overall space footprint of keeping these edits around is never high to cause any concern. Also situation has never occurred where these defaults values needed to be lowered Unnecessary edits (those beyond the retain number configurations) are only purged upon each successful checkpointing at the active namenode, which purges the local NameNode edits files and asks the Journal Nodes to purge their edits file. So if a checkpoint is not occurring, that can cause edits file to be not purged. These two properties are not commonly changed and therefore are not exposed as separate properties within Cloudera Manager, hence they will need to be added in the NameNode Safety Valve ("NameNode Configuration Safety Valve for hdfs-site.xml") and requires Namenode restart. <property> <name>dfs.namenode.num.extra.edits.retained</name> <value>1000000</value> </property> <property> <name>dfs.namenode.max.extra.edits.segments.retained</name> <value>10000</value> </property>   ... View more

@josr89 seems like you have permission issues for below path hdfs://env1/apps/hbase/data/staging Kindly assign proper permission for the same path for userA from the Ranger UI ... View more

Yes @mike_bronson7  above steps also works ... View more

=> If above steps still gives you issues then you can simply execute step 5 or below Cmd from Standby NN // Bootstrap Standby NameNode. This command copies the contents of the Active NameNode's metadata directories (including the namespace information and most recent checkpoint) to the Standby NameNode.   # hdfs namenode -bootstrapStandby Note: Step 1 to step 3 is process of creating new fsimage but if your Active NN is already up and running then I would directly login in to Standby and then perform bootstrapStandby operation ... View more