I finally gave up.. Then last week we were asked to add new nodes to the cluster. I got the newest version of CDH distro and those machines got added to the cluster properly.   I can only assume there is some bug in the other version.     ... View more

These are all desktop machines. All machines were NTP'd before installation of CDH. I will need to look at the data node logs again. Last time I looked I saw nothing unusual. The datanode was talking to the namenode and writing blocks.. I will try and look at this today but may not be until tomorrow.   thanks -- brian   a  -> Thu Feb 20 08:22:31 EST 2014 o -> Thu Feb 20 08:22:31 EST 2014 at       -> Thu Feb 20 08:22:31 EST 2014 av       -> Thu Feb 20 08:22:31 EST 2014 b         -> Thu Feb 20 08:22:31 EST 2014 m     -> Thu Feb 20 08:22:31 EST 2014 be  -> Thu Feb 20 08:22:31 EST 2014 ba     -> Thu Feb 20 08:22:31 EST 2014 bl  -> Thu Feb 20 08:22:31 EST 2014 st  -> Thu Feb 20 08:22:31 EST 2014 bla      -> Thu Feb 20 08:22:31 EST 2014 bac   -> Thu Feb 20 08:22:31 EST 2014 meg    -> Thu Feb 20 08:22:31 EST 2014 blas      -> Thu Feb 20 08:22:31 EST 2014 br        -> Thu Feb 20 08:22:31 EST 2014 bi     -> Thu Feb 20 08:22:31 EST 2014 ro     -> Thu Feb 20 08:22:31 EST 2014 big    -> Thu Feb 20 08:22:31 EST 2014 * note: all hostnames on the left were shortened. ... View more

I just added in a new node and it shows dead within hdsfadmin -report but in service in cloudera manager.     ... View more

I was mistaken that they were the only 2 machines with the largest disks. There are others. The dead machines are optimusprime and megatron. Though the dfsadmin -report says they have 0 capacity this shows they do have capacity. It also shows them in service and dfsadmin report says "Last contact: Wed Dec 31 19:00:00 EST 1969" but according to the chart they seem to have a valid last contact number.   I am going to add another machine to the cluster and see if that comes in as dead as well.     ... View more

I rebuilt all 19 machines (fresh os and fresh cdh). I still have two dead nodes and they are the same ones that were dead before. HA is not on. The only real difference is that these machines have much larger disks than the other 17.   Brian ... View more

Well i ended up doing some things today to try and solve it. At one point I needed to reboot all the servers for an OS change I made. When I did that the CM manager would not start. Turns out the "/etc/cloudera-scm-server" directory was completely missing.   So I ended up reinstalling the CM manager and getting all the nodes re-added and such. I still have the problem. Now I have a node that says it is both dead and alive and a new node that says it is dead.   I suspect since the directory was missing something hokey must have occurred during the initial installation.   So I think I am just going to wipe everything and install it like I have many times in the past.   Brian ... View more

to fix this i just downloaded the files from http://archive.cloudera.com/cm4/redhat/6/x86_64/cm/4.7.2/RPMS/x86_64/ and then put them in /var/cache/yum/x86_64/6/cloudera-manager/packages/     ... View more

It seems like maybe there is a limit to the # of hosts that I am allowed to have active. I ended up deleting the both the dead nodes (the one that was still dead and the one that was rebuilt as alive). Once I did this the newest dead node that appeared today came back as alive.   Is there some setting that puts nodes in standby or some sort of limit to how many nodes can be in a cluster?   Thanks Brian ... View more

Well the parcel went well. All my hosts are the same versions except for my Mixed Host. My mixed host just happens to be my Cloudera Manager.   No change in my dead hosts..   How does one go about fixing a mixed host when it is the cloudera manager?   Thanks Brian ... View more

I finally got my parcels to be recognized. I am distributing them now. will let you know if that solves the issue.     ... View more

Well I have good and bad news.   I deleted one of the dead nodes. I made a new machine with a new hostname/ip that has not been used in over a year. After I added in the new machine I had a different node become a dead node but the new node is an alive node.   So I still have two dead machines, just one of them is a new one that has been alive for a long time.   crazy? Brian ... View more

the mixed cdh is a single node. using rpms. i have not been able to get parcels to download or get recognized when manually placing them. i have used parcels in the past in different cluster but this time not having luck.   I have restarted the NN, Cluster, datanodes, removed roles and re-added roles, rebooted, and any other combo you can think of.   Maybe tomorrow I will build a new system with a new hostname and ip and add it to the cluster and see if that new node has a problem as well. If so then maybe there is some file that is messed up or possibly duplicate hostname/ip on the network (but i think i ruled that out pretty thoroughly tuesday).   Brian   ... View more

I checked the hosts page. There are no duplicate host names or IP Addresses.   There are no errors that I see with the host inspector. There are about 8 machines that I built at the same time (namenode, 2 dead, 6 alive) that have the exact same CDH versions.. There are about 7 other machines that have a slighlty older version of cdh and one node that is mixed.     Validations     Inspector ran on all 17 hosts.     Individual hosts resolved their own hostnames correctly.     No errors were found while looking for conflicting init scripts.     No errors were found while checking /etc/hosts.     All hosts resolved localhost to 127.0.0.1.     All hosts checked resolved each other's hostnames correctly.     Host clocks are approximately in sync (within ten minutes).     Host time zones are consistent across the cluster.     The group oozie is missing on the following hosts:     The group hue is missing on the following hosts:     The user oozie is missing on the following hosts:     The user hue is missing on the following hosts:     No kernel versions that are known to be bad are running.     No performance concerns with Transparent Huge Pages settings.     1 hosts are reporting with MIXED CDH version     There are mismatched versions across the system. See details below for details on which hosts are running what versions of components.     All managed hosts have consistent versions of Java.     All checked Cloudera Management Daemons versions are consistent with the server.     All checked Cloudera Management Agents versions are consistent with the server. ... View more

2.0.0-cdh4.5.0  (Cloudera Express). I do not think HA is an option with this version.   From NN UI. Live Nodes     :    16 (Decommissioned: 0) Dead Nodes     :    2 (Decommissioned: 0) ... View more

hosts file is fine.   this is in the namenode log on the namenode (hadoop-cmf-hdfs1-NAMENODE-xxxx.log.out) the log entry below is when i start the hdfs process on 192.168.129.1.     2014-02-13 17:09:47,638 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* registerDatanode: from DatanodeRegistration(192.168.129.1, storageID=DS-463268316-192.168.129.1-50010-1392235735193, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=cluster5;nsid=2147262545;c=0) storage DS-463268316-192.168.129.1-50010-1392235735193 2014-02-13 17:09:47,639 INFO org.apache.hadoop.net.NetworkTopology: Removing a node: /default/192.168.129.1:50010 2014-02-13 17:09:47,639 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default/192.168.129.1:50010 2014-02-13 17:09:47,684 INFO BlockStateChange: BLOCK* processReport: from DatanodeRegistration(192.168.129.1, storageID=DS-463268316-192.168.129.1-50010-1392235735193, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=cluster5;nsid=2147262545;c=0), blocks: 2, processing time: 0 msecs ... View more

iptables and selinux are all off. let me look into what you said to check and if anything becomes apparent i will report back. thanks ... View more

Well some more information. I found that the files in the /etc/hadoop/conf on these 2 dead nodes were "stock" files. They had no configuration information for my environment.   So I did "Deploy Client Configuration" and the correct conf files were placed in /etc/hadoop/conf.   I then restarted hdfs and no change, still show dead. I then restarted the cluster and no change, they still show dead in dfsadmin -report. But cloudera manager says they are healthy.   Brian ... View more

Thanks Philip. I found this out about 3 hours after I posted this. I had used that same option in the past but appears I have very short term memory.   Thanks again. Brian ... View more