How can i recommission Node managers and Data nodes through REST API?

[#1]

curl -u admin:admin -i -H 'X-Requested-By: ambari' -X POST -d '{
"RequestInfo":{
"context":"Recommission DataNodes",
"command":"RECOMMISSION",
"parameters":{
"slave_type":"DATANODE",
"included_hosts":"c6401.ambari.apache.org,c6402.ambari.apache.org,c6403.ambari.apache.org"
},
"operation_level":{
"level":"HOST_COMPONENT",
"cluster_name":"c1"
}
},
"Requests/resource_filters":[
{
"service_name":"HDFS",
"component_name":"NAMENODE"
}
]
}' http://localhost:8080/api/v1/clusters/c1/requests

curl -u admin:admin -i -H 'X-Requested-By: ambari' -X POST -d '{
"RequestInfo":{
"context":"Recommission NodeManagers",
"command":"RECOMMISSION",
"parameters":{
"slave_type":"NODEMANAGER",
"included_hosts":"c6401.ambari.apache.org,c6402.ambari.apache.org,c6403.ambari.apache.org"
},
"operation_level":{
"level":"HOST_COMPONENT",
"cluster_name":"c1"
}
},
"Requests/resource_filters":[
{
"service_name":"YARN",
"component_name":"RESOURCEMANAGER"
}
]
}' http://localhost:8080/api/v1/clusters/c1/requests

@Artem Ervits .Can i use like this to recommission the data node and nodemanagers? Whcih one has to be excuted first? Datanodes ?

[#2 · Accepted solution]

You are becoming an advanced user of Ambari API. Your use case is definitely valid but i dont have an answer for exact steps or whether its even possible. What you can do is setup network capture with wireshark and see api calls ambari makes when you recommission a node. My guess is that that option is not available in API as you typically comission a node as part of adding a new node. So maybe as a workaround you would want to delete a node completely from cluster after decommissioning and then just add new node. Let us know if you do run netcapture.

[#3]

Thank you @Artem Ervits I would like to give a try with the above curl commands, will post the result.

[#4]

Surprise us 🙂

[#5]

@Artem Ervits Mostly, it wouldn't work because i didn't see any dfs.include and yarn.include parameters in configurations. 🙂

[#6]

Well then decommission, delete, add node is the way to go.

[#7]

@Artem Ervits I tried to recommission by using above curl command. I got the following error..

"status" : 500, "message" : "An internal system exception occurred: Unsupported action RECOMMISSION for Service: YARN and Component: RESOURCEMANAGER"

[#8]

@Artem Ervits I am still wondering, whatever we will do through ambari, we can do them through REST api right? Why REST api is having limitiation for RECOMMISSION? Is there any theoretical reason?

[#9]

@Ram D I say look at the source code, https://github.com/apache/ambari/blob/966f3031d50b0106dfe5d19abbdd1e3b84d1f059/ambari-server/src/mai...

[#10]

Just to follow up on this one. You can recomission DataNodes and NodeManagers with the following calls. Replace the variables as needed. You may need to remove the line breaks.

Node Manager

curl -u admin:admin -H 'X-Requested-By:ambari' -X POST -d
'{  
   "RequestInfo":{  
      "context":"Recomissioning of host '$host': Recomission NodeManger via REST-API",
      "operation_level":{  
         "level":"HOST",
         "cluster_name":"'$clustername'",
         "host_name":"'$host'"
      },
      "command":"DECOMMISSION",
      "parameters":{  
         "slave_type":"NODEMANAGER",
         "included_hosts":"'$host'"
      }
   },
   "Requests/resource_filters":[  
      {  
         "service_name":"YARN",
         "component_name":"RESOURCEMANAGER"
      }
   ]
}' "http://$ambariHost:8080/api/v1/clusters/$clustername/requests"

DataNode:

curl -u admin:admin -H 'X-Requested-By:ambari' -X POST -d
'{  
   "RequestInfo":{  
      "context":"Scipt-based Recomissioning of host '$host': Recomission DataNode via REST-API",
      "operation_level":{  
         "level":"HOST",
         "cluster_name":"'$clustername'",
         "host_name":"'$host'"
      },
      "command":"DECOMMISSION",
      "parameters":{  
         "slave_type":"DATANODE",
         "included_hosts":"'$host'"
      }
   },
   "Requests/resource_filters":[  
      {  
         "service_name":"HDFS",
         "component_name":"NAMENODE"
      }
   ]
}' "http://$ambariHost:8080/api/v1/clusters/$clustername/requests"

[#11]

Thank you very much @Roland Simonis

I had the same problem (reinstalled a slave from scratch, and decommisioned via Ambari both DN and NM) but Ambari doesn't have a GUI option for recommissioning the NM, so the ResourceManager was always denying access. With the API call you posted I easily recommissioned the NM and now everything is working again as expected.

Thanks!

[#12]

In ambari version AMBARI-2.5.1.7 we needed to do another step after recommissioning steps above mentioned. The nodemanger was going to stopped state after the recommissioning call. Needed to change the state from installed to started.

curl -u user:pass -i -H 'X-Requested-By: ambari' -X PUT -d '{"HostRoles": {"state": "STARTED"}}' http://{$ambari_master_host}:{$ambari_master_port}/api/v1/clusters/{$clustername}hosts/{$host}/host_...;

Same steps are followed by the ambari server when doing the recommission. PFA the logs

// Recomission ( Decommission with include host )

14 Feb 2018 13:28:23,479 INFO [ambari-client-thread-75749] AmbariManagementControllerImpl:3984 - Received action execution request, clusterName=xxxx , request=isCommand :true, action :null, command :DECOMMISSION, inputs :{included_hosts=xxxx, slave_type=NODEMANAGER}, resourceFilters: [RequestResourceFilter{serviceName='YARN', componentName='RESOURCEMANAGER', hostNames=[]}], exclusive: false, clusterName :xxxx

14 Feb 2018 13:28:23,488 INFO [ambari-client-thread-75749] AmbariCustomCommandExecutionHelper:967 - Recommissioning NODEMANAGER and marking Maintenance=OFF on xxx

14 Feb 2018 13:28:32,906 INFO [qtp-ambari-agent-75726] HeartBeatHandler:332 - HeartBeatHandler.sendCommands: sending ExecutionCommand for host xxx role RESOURCEMANAGER, roleCommand CUSTOM_COMMAND, and command ID 25366-0, task ID 45097

// Ambari changes state from Started to Installed

14 Feb 2018 13:28:44,295 INFO [ambari-heartbeat-processor-0] HeartbeatProcessor:609 - State of service component NODEMANAGER of service YARN of cluster xxx has changed from STARTED to INSTALLED at host xxxx according to STATUS_COMMAND report

14 Feb 2018 13:30:29,889 INFO [ambari-client-thread-75692] AbstractResourceProvider:939 - Received a updateHostComponent request, clusterName=xxx , serviceName=YARN, componentName=NODEMANAGER, hostname=xxxxx request={ clusterName=xxxx , serviceName=YARN, componentName=NODEMANAGER, hostname= xxxx , publicHostname=null, desiredState=STARTED, state=null, desiredStackId=null, staleConfig=null, adminState=null, maintenanceState=null}

// Ambari changes state again from Started to Installed

14 Feb 2018 13:30:29,890 INFO [ambari-client-thread-75692] AbstractResourceProvider:650 - Handling update to host component, clusterName=xxx, serviceName=YARN, componentName=NODEMANAGER, hostname=xxxxx , currentState=INSTALLED, newDesiredState=STARTED

14 Feb 2018 13:30:29,910 INFO [ambari-action-scheduler] ServiceComponentHostImpl:1028 - Host role transitioned to a new state, serviceComponentName=NODEMANAGER, hostName=xxxx oldState=INSTALLED, currentState=STARTING