Any suggestion to fix this issue.. 🙂 @Trinity wrote: Hi Gautam,   Thanks for your quick response, please find my answers below.   "HDFS Under replicated blocks" implies that some blocks are not duplicated enough to satisfy the default replication factor of 3. If possible consider setting up clusters with at least 3 nodes.           As of now our requirement donot need to have 3 nodes.    "Missing Blocks" implies the datanodes which had block before shutdown now don't have it when they booted up. This could happen with the Instance Store. What kind of storage did you use on the nodes? This is explained here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Storage.html         We are configured the entire environment in EBS volumes. Our working scenario is, Cluster with 2 nodes After making changes we need to shutdown the instances (since the application is in the developement stage). When we need to perform the developement we will be starting the cluster and perfom the changes.       This missing block is showing when we are starting the cluster after shutting down the same for a duration of 2 - 3 days   When you run "hadoop fsck -delete" you are telling the namenode to delete files whose blocks cannot be located. This is fine for temporary files. Before running it however you should run "hdfs fsck -list-corruptfileblocks", identify the reason why the blocks are missing. If the blocks are recoverable, you won't have to delete the files themselves.          The Hbase won't be starting without executing "hadoop fsck -delete" command and the "hdfs fsck -list-corruptfileblocks" out shows around 105 missing blocks. The missing block navigation (path to the block) showing the date stamp of the time of shutdown. Is that means we are not allowed to shutdown and start the cluster according to our requirement?    "could only be replicated to 0 nodes, instead of 1" could mean the datanodes are not healthy. Check the datanode logs under /var/log/hadoop-hdfs on both nodes to see what the problem might be. If it's not clear, paste the relevant parts to pastebin and ​​give us the URL​          This error is happening after running the command "hadoop fsck -delete" . After this command executaion the Hbase will be starting up and the HDFS will be showing the error "could only be replicated to 0 nodes, instead of 1"     Our ultimate goal is create a cluster with 2 nodes Shutdown cluster after completing my tasks Start the cluster when ever we need to make changes or demo purpose. Please let us know is the above scenario is possible or not in CDH 5.X.X.   Thanks in advance Akash.    ... View more

Hi Gautam,   Thanks for your quick response, please find my answers below.   "HDFS Under replicated blocks" implies that some blocks are not duplicated enough to satisfy the default replication factor of 3. If possible consider setting up clusters with at least 3 nodes.           As of now our requirement donot need to have 3 nodes.    "Missing Blocks" implies the datanodes which had block before shutdown now don't have it when they booted up. This could happen with the Instance Store. What kind of storage did you use on the nodes? This is explained here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Storage.html         We are configured the entire environment in EBS volumes. Our working scenario is, Cluster with 2 nodes After making changes we need to shutdown the instances (since the application is in the developement stage). When we need to perform the developement we will be starting the cluster and perfom the changes.       This missing block is showing when we are starting the cluster after shutting down the same for a duration of 2 - 3 days   When you run "hadoop fsck -delete" you are telling the namenode to delete files whose blocks cannot be located. This is fine for temporary files. Before running it however you should run "hdfs fsck -list-corruptfileblocks", identify the reason why the blocks are missing. If the blocks are recoverable, you won't have to delete the files themselves.          The Hbase won't be starting without executing "hadoop fsck -delete" command and the "hdfs fsck -list-corruptfileblocks" out shows around 105 missing blocks. The missing block navigation (path to the block) showing the date stamp of the time of shutdown. Is that means we are not allowed to shutdown and start the cluster according to our requirement?    "could only be replicated to 0 nodes, instead of 1" could mean the datanodes are not healthy. Check the datanode logs under /var/log/hadoop-hdfs on both nodes to see what the problem might be. If it's not clear, paste the relevant parts to pastebin and ​​give us the URL​          This error is happening after running the command "hadoop fsck -delete" . After this command executaion the Hbase will be starting up and the HDFS will be showing the error "could only be replicated to 0 nodes, instead of 1"     Our ultimate goal is create a cluster with 2 nodes Shutdown cluster after completing my tasks Start the cluster when ever we need to make changes or demo purpose. Please let us know is the above scenario is possible or not in CDH 5.X.X.   Thanks in advance Akash.  ... View more

Hey Gautam,   Thanks for the quick response. Please find my responses below.   "Missing Blocks" implies the datanodes which had block before shutdown now don't have it when they booted up. This could happen with the Instance Store. What kind of storage did you use on the nodes? This is explained here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Storage.html        I have configured using EBS instead of instance store. We need to shut down the instance after the usage (since the application is not yet expossed to live). My working scenatio is Configured a cluster with 2 nodes.  Shutdown them after completing my jobs on there. Start once I want to do some changes in the cluster. In my case, issue is once I start the cluster (instances) after 2 - 3 days (after shutting down) there will be showing the errors of missing blocks. Will shutdown and starting the server according to our use makes any issue in CDH5.x.x?   When you run "hadoop fsck -delete" you are telling the namenode to delete files whose blocks cannot be located. This is fine for temporary files. Before running it however you should run "hdfs fsck -list-corruptfileblocks", identify the reason why the blocks are missing. If the blocks are recoverable, you won't have to delete the files themselves.        Ok, but the HBASE wont comming up with out resolving this missing block issue. Is there any other method to fix this missing block?    "could only be replicated to 0 nodes, instead of 1" could mean the datanodes are not healthy. Check the datanode logs under /var/log/hadoop-hdfs on both nodes to see what the problem might be. If it's not clear, paste the relevant parts to pastebin and ​​give us the URL​         This happens after running the "hadoop fsck -delete" command   ... View more