Member since
09-03-2020
341
Posts
8
Kudos Received
13
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 835 | 01-10-2026 11:06 PM | |
| 592 | 01-10-2026 10:40 PM | |
| 722 | 01-10-2026 12:24 AM | |
| 556 | 01-09-2026 09:51 PM | |
| 463 | 01-09-2026 09:41 PM |
01-09-2026
09:41 PM
@Maddy2 FYI ➤ Based on the logs you provided, your NameNode is failing to start because it has encountered a metadata inconsistency while replaying the Edit Logs. This is a critical issue where the NameNode's current state (from the FSImage) contradicts the instructions in the Edit Logs it is trying to process. ➤ The Root Cause The specific error is a java.lang.IllegalStateException during an OP_MKDIR operation (Transaction ID: 29731504). The NameNode is trying to create a directory (/tmp/hive/nifi/...), but the checkState fails because the parent directory for that path does not exist in the namespace it just loaded from the FSImage. This likely happened because: Disk Expansion/Reboot Out of Sync: When you expanded the disk and rebooted, one of the storage directories (/mnt/resource/hadoop/hdfs/namenode) was flagged as unformatted or empty. Metadata Corruption: There is a mismatch between your last successful checkpoint (fsimage_0000000000029731317) and the subsequent edits stored in your Journal Nodes. ➤ Recommended Solution: Metadata Recovery Since this is an HDP (Hortonworks Data Platform) cluster with High Availability (HA), you should attempt to recover by syncing from the "good" metadata or forcing a metadata skip. => Step 1: Identify the Healthy NameNode Ensure you are working on the NameNode that has the most recent and intact data. Check the other NameNode's logs to see if it also fails at the same Transaction ID. => Step 2:On Standby or failing Namenode kindly Check the permission of edits log and fsimage present in path dfs.namenode.name.dir and see if it matches with permission mentioned in Active Namenode =>Step 3: Bootstrap from the Standby (If HA is healthy) If one NameNode is able to start or has better metadata, you can re-sync the failing node: => Stop the failing NameNode. On the failing node, clear the NameNode storage directories (as defined in dfs.namenode.name.dir). Run the bootstrap command to pull metadata from the active/healthy NameNode: $ hdfs namenode -bootstrapStandby 4. Start the NameNode.
... View more
01-09-2026
07:51 AM
Thanks for the suggestion, I will go for distcp because we have hundred thousand of files and "only" several thousand of them must be restored
... View more
12-12-2024
10:08 AM
1 Kudo
@irshan When you add balancer as a role in the HDFS cluster, it indeed will show as not started. So its an expected one. Coming to your main query, it could be possible that when you run the balancer, the balancer threshold could be with in the default percentage of 10, so it won't move the blocks. You may have to reduce the balance threshold and try again.
... View more
05-01-2024
03:59 AM
@kpalanisamy ➤ We also have a alternate hbase shell native approach through which we can determine the RegionName and RegionServer from the rowkey $ locate_region 'namespace:tablename','rowkey' HOST REGION Regionserver-name:16 {ENCODED => regionName, NAME => 'namespace:tablename,rowkey.regionName.', STARTKEY => 'f0046', ENDKEY => 'f0245cf'} 1 row(s) Took 0.6760 seconds => #<Java::OrgApacheHadoopHbase::HRegionLocation:0x4070c4ff>
... View more
04-09-2024
05:59 AM
✥ In CDH6 HBase, the property was removed per https://issues.apache.org/jira/browse/HBASE-15989 because we allow all altering operations without disabling table. cc : @webtube
... View more
04-06-2024
12:35 AM
1 Kudo
✥ For the Error "Unrecognized option:-j" kindly manually type the keyword -j instead of copy-paste => Note: you generally notice such exceptions when you have special character pasted ✥ Kindly make sure you first bypass the stuck procedure and locks $ hbase hbck -j /tmp/target/hbase-hbck2-1.3.0-SNAPSHOT.jar bypass -o -r <pid> => For reviewing the stuck procedure kindly navigate to Hmaster ui => Procedure & locks and first bypass the proc id visible in lock section ✥ The you can consider closing the region State before disabling table $ hbase hbck -j /tmp/target/hbase-hbck2-1.3.0-SNAPSHOT.jar setRegionState $i CLOSED $ hbase hbck -j /tmp/target/hbase-hbck2-1.3.0-SNAPSHOT.jar setTableState <tablename> DISABLED => Note: Make sure the 3k regions are all part of the same table " " which you wish to disable/remove ✥ Once table is disabled then you can login into hbase shell and then perform drop table operation $ drop 'tablename'
... View more
03-25-2024
02:51 AM
1 Kudo
Anyone had this problem before ?
... View more
03-20-2024
10:54 PM
1 Kudo
Hi @josr89 you can provide WRITE access to user "userA" to the below path under the "cm_hdfs" repository in Ranger, wait for plugins to sync and then rerun the operation. path: /apps/hbase/data/staging
... View more
02-27-2024
12:29 AM
2 Kudos
Yes @mike_bronson7 above steps also works
... View more
02-05-2024
05:21 AM
1 Kudo
=> If above steps still gives you issues then you can simply execute step 5 or below Cmd from Standby NN // Bootstrap Standby NameNode. This command copies the contents of the Active NameNode's metadata directories (including the namespace information and most recent checkpoint) to the Standby NameNode. # hdfs namenode -bootstrapStandby Note: Step 1 to step 3 is process of creating new fsimage but if your Active NN is already up and running then I would directly login in to Standby and then perform bootstrapStandby operation
... View more
- « Previous
-
- 1
- 2
- Next »