Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

can't delete bad node from the cluster

avatar

64-node cluster

one node is bad..no longer communicates 

I want to remove him from the cluster

 

ch-8 10.71.0.108 /default CDH 5 Cluster 1

2 Role(s)

Good Health 12.38s ago
ch-9 10.71.0.109 /default Unknown Cluster 1

2 Role(s)

Bad Health None

 

here he is in the hosts. he's a data node and a nodemanager (yarn)

 

When I try to delete, it tells me

Delete Hosts The following 1 host(s) cannot be deleted because they have role instances or are not completely decommissioned:

Host Role Instances
ch-9 nodemanager (ch-9) and 1 other role(s).

If I try this, it doesn't work Remove Hosts From Cluster

Removing these hosts will stop and delete all roles running on them and then remove them from their clusters. The hosts will still be managed by Cloudera Manager and can be utilized after being added to new or existing clusters.
Role data directories will not be deleted.

Host Role

ch-9 NodeManager, DataNode
Decommission Roles (Warning: Removing the hosts without decomissioning the roles running on them can result in permanent data loss.) Skip Management Roles   
 
 
Command Details: Hosts Decommission
Command Context Status Started at Ended at
Hosts Decommission Finished Mar 31, 2014 1:09:28 PM PDT Mar 31, 2014 1:09:28 PM PDT
Command 'DecommissionWithWait' failed for service 'yarn'
     
Child Commands
All   Failed Only   Active Only
Command (Child commands) Context Status Started at Ended at
Decommission (2) YARN (MR2 Included) Finished Mar 31, 2014 1:09:28 PM PDT Mar 31, 2014 1:09:28 PM PDT
Failed to perform decommission.


 
 

 Basically, if I can't talk to the node, I can't stop/decommission/delete him How should I do it?

1 REPLY 1

avatar
Expert Contributor

Please check the permission for the nodemanager directory. The owner or group must be yarn.

Then try to decommission and it will distribute the blocks to other datanodes and then will decommission.