A quick search in the code base tells me that we have these following policies AvailableSpaceBlockPlacementPolicy BlockPlacementPolicyDefault BlockPlacementPolicyRackFaultTolerant BlockPlacementPolicyWithNodeGroup BlockPlacementPolicyWithUpgradeDomain > yet didn't find any documentation listing the available choices. You are absolutely right, we can certainly do better on documenting this, Thanks for pointing this out. I will address this in an Apache JIRA. ... View more

@Malay Sharma Really depends on your workload. The best case is if you can get good block sizes, that if you have really large files, you should set your block size to 256MB( this reduces the number of blocks in the file system, since you are storing more data per block), or use the default 128 MB. If you have a running cluster, using smart sense, you can get a view of the file size distribution of your current cluster. That will give you an idea if you need to tune the block size for performance. From experience, I can tell you that the performance of your cluster is not going to be dependent on block size unless you have a large number of files in the system. Also, unlike a physical file system, setting the block size to 128MB does not mean that each block write will use up 128MB. HDFS will only use the number of bytes actually written, so there is no waste of space because of the block size. ... View more

@Malay Sharma HDFS is very good at caching all file names and block addresses in memory. This is done at the Namenode level. This makes HDFS incredibly fast. So if you make modifications to the file system or read a file location, all of this can be served with no disk I/Os. This design choice of keeping all metadata in memory at all times has certain trade-offs. One of them is that we need to spend a couple of bytes (think 100s of bytes) per file and blocks. This leads to an issue in HDFS -- when you have a file system with 500 million to 700 million -- the amount of RAM that needs to be reserved by the Namenode becomes large. Typically, in sizes of 256 GB or more. At this size, the JVM is hard at work too; since it has to do things like garbage collection. There is also another dimension to this when you have 700 million files, it is quite possible that your cluster is serving 30-40K or more requests per second. This also creates lots of memory churn. So a large number of files, combined with lots of file system requests makes Namenode a bottleneck in HDFS or in other words, the metadata that we need to keep in memory creates a bottleneck in HDFS. There are several solutions / work in progress to address this problem -- HDFS federation -- That is being shipped as part of HDP 3.0, allows many names nodes to work against a set of Datanodes. In HDFS-7240, where we are trying to separate the block space from the namespace, that allows us to immediately double or quadruple the effective size of the cluster. Here is good document that tracks various issues and different approaches to scaling the namenode -- Uber scaling namenode There is another approach where we try to send the read workloads to the secondary namenode, freeing up the active namenode and thus scaling it better. That work is tracked in Consistent Reads from Standby Node Please let me know if you have any other questions. Thanks Anu ... View more

@Michael Bronson Assuming you are talking about a data node. If you are able to replace this disk trivially, that is the operation is simply to pull the disk out of a JBOD, then you can shut down this datanode, replace the disk, format and mount it back. HDFS will detect that it has lost a set of blocks (probably it has already done it since the disk is faulty and no i/o is happening to that disk), and replicate them correctly. You can check if you have any under-replicated blocks in your cluster. You can replace the disk and things will return to normal. There is, however, a small hitch, this new disk will not have the same amount of data as the other disks. If you are running Hadoop 3.0-- it is still in beta and not production ready -- you can run the diskBalancer tool, which will move data from other disks to this new disk. Generally, this will not be an issue. If the disk replacement in your machines is not as straightforward as I described, you can ask Ambari to put this machine into a maintenance state. That will tell HDFS not to replicate all the blocks after the 10 mins window ( by default) when a machine is declared dead. You can do that and then do the operation. Just so that you are aware, HDFS supports a notion of failed volumes. So if you have a data node with a large number of disks, say 8 disks, then you can set the failed volume tolerance, say to something like 2. This will make sure that node works well even in the face of two disks failing. if you do that, you can replace disks when you have a scheduled maintenance window with downtime. Please let me know if you have any more questions or need more help on this. ... View more

@ARUN it is very dependent on the type and nature of your data. It is also influenced by the applications that consume data. ... View more

@Michael Bronson Just to add to what @Xiaoyu Yao said; if you want to clean only a data node (worker23), you will need to ssh to that node. Then you can either format the data disks or delete data. You can see in Ambari which paths map to the data store. But you need to make sure that are actual disks if you want to run format. A safer approach would be to run "rm -rf /data/disk1", "rm -rf /data/disk2", assuming data node has stored data in a path called /data/disk1 and /data/disk2 Just as @Xiaoyu Yao mentioned, please do NOT format the namenode, the whole cluster will be lost. ... View more

Since you said only one node, I am not sure if you mean there is only one physical machine in the cluster. if that is the case, you want to enable something called pseudo-distributed mode, where all processes, that is namenode, datanode etc. are going to run on the same machine. You can find instructions here. https://hortonworks.com/hadoop-tutorial/introducing-apache-ambari-deploying-managing-apache-hadoop/ On the other hand, if you want to run namenode on a single machine -- but you have a set of physical nodes, you can use Ambari normally. if you want to use Amabri, You can select HDFS and the standard installation does not assume that you have HA. If you want to turn on HA then use HA Wizard. ... View more

@Tomomichi Hirano No you cannot. Block reports serve an essential function that allows Namenode to reconcile the state of the cluster. That is these block reports tell namenode what blocks are needed and what blocks are to be deleted, if the block is under replicated etc. etc. Full block reports are expensive for Namenode process( there are both incremental and full reports) so it is set to a longer value. However, if you set it to really long values like 1 month, your namenode might not work correctly. Typically the only reason to change this value is if your Namenode is under severe load, so if you are not experiencing such a problem, I would suggest that you don't change this parameter. ... View more

@Namit Maheshwari yes, it is possible - couple of ways you can do this are: 1. You can do it from JAVA code by using JMX Apis. 2. You can connect to namenode:httpport;/jmx and get a json based response. 3. You can configure a Namenode/Datanode Metrics log file which will dump into a file which you can parse I am probably missing some more ways :), hope this helps you. ... View more