I have built a 6 node cluster for a Proof of Concept on AWD using Director. The intent of this set of tests is to determine if we can build a Cloudera cluster with Director, run a series of jobs, use CM and EC2 to stop all the servers when done and then restart them when further jobs need to be executed, i.e.. A predefined lab cluster that gets charged only when in use. I am having problems with corrupt hdfs files on restart. I am tryng to pinpoint the right place to look for clues. Is it a 'connection' btween Director and the Cluster (something defined when the cluster was built) or something about using EBS and restating the servers. Any ideas or place to look for clues would be helpful. Here is some background: Setup : (1 Director, 1 CM, 1 Master, 3 Workers and 1 Gateway). I am using CM/CDH 5.4.7. CM and Master are m3.xlarge. Gateway and Director are m3.medium Workers are m3.large. I built a new AMI that is based on ami-8767d1ec, but that I have upgraded to Java 1.8_45. The root drive has ben adjutsed to use 250 GB EBS. Problem: The original creation is built and starts fine. CM shows 100% green. We can add data, run mapreduce jobs on Yarn, examine results, etc. I then stop the CM cluster services, stop the CM Management services, and then go into EC2 to stop all servers. I then restart the machines, go into CM server and restart CM cluster and managment services. All servces comeup fine initially with the exception of HDFS, which starts with an error on the Canary test. Then eventually HDFS goes into safemode and Hive also triggers a metastore canary test error and both services are flagged red and the Namenode goes down. Leaving safemode and restarting the Namenode brings me back to the original error, but eventully it finds itself back to safemode wothout a namenode. Here are some messages from the Namenode UI: Safe mode is ON. The reported blocks 193 needs additional 192 blocks to reach the threshold 0.9990 of total blocks 385. The number of live datanodes 3 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached. 878 files and directories, 385 blocks = 1263 total filesystem object(s). Heap Memory used 166.71 MB of 990.75 MB Heap Memory. Max Heap Memory is 990.75 MB. Non Heap Memory used 46.81 MB of 47.38 MB Commited Non Heap Memory. Max Non Heap Memory is 130 MB. I examined the missing blocks and the majority are blocks that are there via CDH: Oozie directories, etc. Any ideas on how I might track this down? Thanks - rd
... View more
