One of my datanodes' local data disks failed and are in some wierd state. The HDFS activity was going into the directory of the mount points in the local file system which caused it to fill to 98%. The data node wasnt dead but could not decommision, remove him or delete him from the "hosts" menu. Every service except the HDFS service could be brought down. The message I keep getting even now, is "Pending Command". The way I deleted the node was to "delete" from each service. Now, I get the Stale Configuration message, I try to deploy the configuration, I get the "pending Command" message or this from the topology-map
I shut the cluster down, rebooted the cloudera manager, restarted the cluster. The problem went away. The vestiges of the old server is gone and I have successfully done the client deployment as per CM request.