We have seen performance issue and stability issue in heavy workload system, this is due to disk latency/Shared Disk. I.e frequent namenode failover, longer boot time, slower checkpoint, slower logging, Higher fsync will cause session expiry, etc. We never recommend a shared disk for Namenode, Journal node and Zookeeper. All of these services should have a dedicated disk. You can configure following disk type according to your HDFS workload. [dfs.namenode.name.dir] Namenode fsimage directory, dedicated disk => HDD 15K RPM [dfs.journalnode.edits.dir] Namenode edit log directory, dedicated disk => SSD [dataDir from zoo.cfg] Zoookeeper snapshot and transaction logs, for normal usage from ZKFC and HBase => HDD 15K RPM Zoookeeper snapshot and transaction logs, If it used by Nifi/ Accumulo/ Kafka/ Storm/ HBase and ZKFC => SSD If you are using RAID for meta-directory(dfs.namenode.name.dir & dfs.journalnode.edits.dir), then disable RAID and check the Non-RAID performance. There is a strong redundant for meta-directory (fsimage & edits are available from Standby NameNode and remaining QJNs). If RAID is not disabled for JN, then consider using different RAID. RAID 1 and RAID 10 are also good for the dfs.journalnode.edits.dir set rather than RAID 5, due to increased in write latency for small block writes. if you don't have a faster disk, then don't consider using fsimage replication. It will impact write performance even if one of the disks slower. ... View more

The service RPC port gives the DataNodes a dedicated port to report their status via block reports and heartbeats. The port is also used by Zookeeper Failover Controllers for periodic health checks by the automatic failover logic. The port is never used by client applications hence it reduces RPC queue contention between client requests and DataNode messages. Steps: 1) Ambari -> HDFS -> Configs -> Advanced -> Custom hdfs-site -> Add Property dfs.namenode.servicerpc-address.<dfs.internal.nameservices>.nn1=<namenode1 host:rpc port> dfs.namenode.servicerpc-address.<dfs.internal.nameservices>.nn2=<namenode2 host:rpc port> dfs.namenode.service.handler.count=(dfs.namenode.handler.count / 2) This RPC port receives all DN and ZKFC requests like block report, heartbeat, liveness report, etc.. Example from hdfs-site.xml, dfs.nameservices=shva dfs.internal.nameservices=shva dfs.ha.namenodes.shva=nn1,nn2 dfs.namenode.rpc-address.shva.nn1=hwxunsecure2641.openstacklocal:8020 dfs.namenode.rpc-address.shva.nn2=hwxunsecure2642.openstacklocal:8020 dfs.namenode.handler.count=200 Service RPC host, port and handler threads: dfs.namenode.servicerpc-address.shva.nn1=hwxunsecure2641.openstacklocal:8040 dfs.namenode.servicerpc-address.shva.nn2=hwxunsecure2642.openstacklocal:8040 dfs.namenode.service.handler.count=100 2) Restart Standby Namenode. You must wait till Standby Namenode out of safemode. Note: You can check Safemode status in Standby Namenode UI. 3) Restart Active Namenode. 4) Stop Standby Namenode ZKFC controller. 5) Stop Active Namenode ZKFC controller. 6) Login Active Namenode and reset Namenode HA state. #su - hdfs $hdfs zkfc -formatZK 7) Login Standby Namenode and reset Namenode HA state. #su - hdfs $hdfs zkfc -formatZK 😎 Start Active Namenode ZKFC controller. 9) Start Standby Namenode ZKFC controller. 10) Rolling restart the Datanodes. Note: Please check, Nodemanager should not be installed in Namenode box because it uses same port 8040. If installed then you need to change service RPC port from 8040 to different port. Ref: scaling-the-hdfs-namenode ... View more

Missing Block Mark missing if all of the block replicas of that file is not reported to Namenode. Corrupt Block Mark corrupt if all of the block replicas of that file is corrupted (Or) none of them are reported to Namenode. The checklist must be done before you confirm block is corrupted/missing. Check if all datanodes are running in the cluster Check if you see dead datanodes Check if disk failure from multiple datanode Check if disk out of space from multiple datanode Check if block report is rejected by namenode (It can be seen from namenode log as a warning/error) Check if you changed any config groups Check if block physically exists in local filesystem or removed by users unknowingly. Ex: "find <dfs.datanode.data.dir> -type f -iname <blkid>*". Repeat the same step in all datanodes Check if too many blocks hosted in a single datanode Check if block report fails with "exceeding max RPC size", default 64 MB. You can see this warning from namenode log "Protocol message was too large. May be malicious" Check if mount point is unmounted because of filesystem failure Check if block is written into root volume because of disk auto unmount. Data might be hidden if you remount the filesystem on top of existing datanode dir. Note: You will lose data if you run "hdfs fsck / -delete". Please ensure you have done all checklist ... View more

Simple API to find region name for given row key. It will also give you hint about the region - Where key will be placed. To compile class: $JAVA_HOME/bin/javac -cp `hbase classpath`: Regionfinder.java To run tool: $JAVA_HOME/bin/java -cp `hbase classpath`: Regionfinder <tablename> <rowkey> Regionfinder.java import java.io.IOException; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hbase.HBaseConfiguration; import org.apache.hadoop.hbase.HRegionLocation; import org.apache.hadoop.hbase.TableName; import org.apache.hadoop.hbase.client.Connection; import org.apache.hadoop.hbase.client.ConnectionFactory; import org.apache.hadoop.hbase.client.Get; import org.apache.hadoop.hbase.client.RegionLocator; import org.apache.hadoop.hbase.client.Result; import org.apache.hadoop.hbase.client.Table; import org.apache.hadoop.hbase.util.Bytes; public class Regionfinder { private static void usage(String[] args) { if (args.length < 2) { System.out.println("$JAVA_HOME/bin/java -cp `hbase classpath`: Regionfinder <tablename> <rowkey>"); System.exit(-1); } } public static void main(String[] args) throws IOException { usage(args); TableName tablename = TableName.valueOf(args[0]); String rowkey = args[1]; byte[] keyInBytes = Bytes.toBytes(rowkey); Configuration conf = HBaseConfiguration.create(); Connection connection = ConnectionFactory.createConnection(conf); Table table = connection.getTable(tablename); RegionLocator regionLocater = connection.getRegionLocator(tablename); HRegionLocation regionLocation = regionLocater.getRegionLocation(keyInBytes); Result result = table.get(new Get(keyInBytes)); if(result.isEmpty()){ System.out.println("Rowkey "+rowkey+" is not exist in any region. It will be placed in region : "+regionLocation.getRegionInfo().getRegionNameAsString()); }else{ System.out.println("Table Name = " + tablename + "\n" + "Row Key = " + rowkey + "\n" + "Region Server = " + regionLocation.getServerName() + "\n" + "Region Name = " + regionLocation.getRegionInfo().getRegionNameAsString() + "\n" + "Encoded Region Name = " + regionLocation.getRegionInfo().getEncodedName()); } } } Ex Output: #java -cp `hbase classpath`: Regionfinder sp 100 Table Name = sp Row Key = 100 Region Server = hwx2633.openstacklocal,16020,1528109888044 Region Name = sp,100,1521182497105.6e87f8a4f3bf7c2762d644dba8e98022. Encoded Region Name = 6e87f8a4f3bf7c2762d644dba8e98022 Non existing row key: #java -cp `hbase classpath`: Regionfinder sp 10 Rowkey 10 is not exist in any region. It will be placed in region : sp,10,1521182497105.ecc8568cb94776ad83ee04cbb422bff0 ... View more

There are two sections which illustrate moving the Journal directory on the same Host. You can follow one of the section according to your need. Section 1: ( Service Downtime is not required ) 1. Change the settings in Ambari for journal node edits directory(dfs.journalnode.edits.dir) from /hadoop/hdfs/journal/ to /data/1/journal/ 2. Don't restart any services immediately. 3. Stop the journal node on NODE1. a. SSH to NODE1 b. sudo mkdir -p /data/1/journal c. sudo chown hdfs:hadoop /data/1/journal d. sudo rsync -ahvAX /hadoop/hdfs/journal/* /data/1/journal/ Start the journal node on NODE1 4. Repeat step 3 for remaining two journal nodes NODE2 and NODE3. 5. Restart the required services accordingly (Rolling or All at once) Section 2: ( Service Downtime is required ) 1. Create the new directory on all Journal nodes. Example: # mkdir -p /data/1/journal/ 2. Change ownership to "hdfs:hadoop" on all Journal nodes. Example: # chown hdfs:hadoop /data/1/journal/ 3. Backup an exiting "Fsimage and Edits" from " dfs.namenode.name.dir", "dfs.journalnode.edits.dir" directory on both Namenode and all Journal nodes. 4. Active Namenode - Perform the saveNamespace which will Merge "FSImage and Edits". # su - hdfs $ hdfs dfsadmin -safemode enter $ hdfs dfsadmin -saveNamespace 5. Stop Active Namenode and Standby Namenode. 6. Stop all Journal nodes. 6. Update, latest Journal node directory. Ambari -> HDFS -> Configs -> Advanced -> Advanced hdfs-site -> dfs.journalnode.edits.dir 7. Start all Journal nodes. 8. Format JN directory. # su - hdfs $ hdfs namenode -initializeSharedEdits Note: The command should be run on one of the Namenode hosts. 9. Start Active Namenode. Note: Don't proceed next step until Namenode out of Safemode. 10. Bootstrap Standby Namenode. # su - hdfs $ hdfs namenode -bootstrapStandby Note: The command should be run on Standby Namenode. During, You will get the option to format the Storage directory. Press Y and hit enter. Re-format filesystem in Storage Directory /hadoop/hdfs/namenode ? (Y or N) Y 11. Start Standby Namenode. ... View more