@Namit Maheshwari yes, it is possible - couple of ways you can do this are: 1. You can do it from JAVA code by using JMX Apis. 2. You can connect to namenode:httpport;/jmx and get a json based response. 3. You can configure a Namenode/Datanode Metrics log file which will dump into a file which you can parse I am probably missing some more ways :), hope this helps you. ... View more

@slachterman Thanks for sharing your experience. @Gerd Koenig Sorry to hear that this happening quite often. This might be an issue in Ranger as mentioned by @slachterman If you have enough details, please feel free to open an Apache Ranger JIRA so that Ranger team gets a chance to look at this. ... View more

@Huahua Wei I am going to presume that the intend of you question is to reduce the downtime of Namenode while you have HA for the clients. I will answer that first and follow up with the specific answer to SafeMode question. The whole point of HA is to minimize the downtime of Namenodes. Here is what you should do. You should fail over to the Standby namenode and generally it should be very fast, maximum couple of retries from clients and they will be able to continue working against the cluster without any detectable loss of availability. That is, most of your YARN jobs should be fine. If this is not happening, some of your configs for Zookeeper or HDFS is not optimal. If you are specifically asking about Safemode, Safemode is used to make sure that enough datanodes and blocks from those datanodes have reported in. You can set a lower threshold for this -- The namenode when it meets that threshold will exit the safemode. There is a slider for this threshold in the Ambari UI for HDFS settings. If you really want to tell namenode to ignore this whole waiting for datanodes and block reports -- you are free to do so, there is a command line option which allows you to run "Safemode exit". Both of these are not really good things for your cluster. Spending a bit of time when the cluster is starting up for the first time is actually a good idea, so that you know that your data and nodes are in good shape. But yes, you can reduce the Safemode waiting time via config or via command line. ... View more

@PJ Depending on your situation there are many solutions. This was a fundamental issue in HDFS for a long time. Very recently we have fixed this issue, there is a new tool called DiskBalancer -- which ships with trunk. If you want to see the details of the design and fix -- please look at https://issues.apache.org/jira/browse/HDFS-1312 . It essentially allows you to create a plan file -- That describes how data will be moved from disk to disk and then you can ask a datanode to execute it. Unfortunately, this tool is yet to be shipped as part of HDP. Soon we will be shipping it. Presuming you are running an older version of HDFS, and you have many datanodes in the cluster, you can decommission this full node and re-add them. However the speed of HDFS replication is throttled, so if you want this to happen fast, you might have to set these parameters in your cluster. dfs.namenode.replication.work.multiplier.per.iteration = 10 dfs.namenode.replication.max-streams = 50 dfs.namenode.replication.max-streams-hard-limit = 100 Last, the one that I would least advise you to do, I am writing this down for the sake of completeness, is to follow what the apache documentation suggests. https://wiki.apache.org/hadoop/FAQ#On_an_individual_data_node.2C_how_do_you_balance_the_blocks_on_the_disk.3F Please note, this is a dangerous action and unless you really know what you are doing this can lead to data loss. So please, please make sure that you can restore the machine to earlier state if you really decide to go this route. ... View more

This issue is fixed in Apache. https://issues.apache.org/jira/browse/HDFS-9184 . However this might be in a future apache release as this has been fixed very recently. ... View more