Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cannot add hosts via the API - heartbeat fails

Highlighted

Cannot add hosts via the API - heartbeat fails

New Contributor

I am trying to automate a cluster scale up action. I'm trying to add a newly provisioned node to an existing cluster via the REST API.

I am installing ambari-agent on the new host before hand using chef. After that I follow the steps mentioned here --> https://cwiki.apache.org/confluence/display/AMBARI/Add+a+host+and+deploy+components+using+APIs

- After ambari agent comes up, I can see the new host in http://ambari-server:8080/api/v1/hosts.

- After running the command to add host to cluster, I get this:

```

Sending request to add node to cluster vsatokar_23...

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
HTTP/1.1 201 Created
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Cache-Control: no-store
Pragma: no-cache
Set-Cookie: AMBARISESSIONID=48sbto3p5bno1rwe3o7fhfiqp;Path=/;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
User: admin
Content-Type: text/plain
Content-Length: 0

```

- Later, I do see the new host in http://ambari-server:8080/api/v1/clusters/mycluster/hosts/hostfqdn

But now the problem is that the server which is added shows the host status as UNKNOWN

"host_health_report" : "",
    "host_name" : "fqdn",
    "host_state" : "HEALTHY",
    "host_status" : "UNKNOWN",

- It then doesn't allow me to add any components to this host.

I'm stuck with this for days now..any help/pointers will be much appreciated!

Note:

- selinux, iptables are disabled.

- ambari-agent.ini has the correct entries

- I've tried restarting the ambari-server, and agent to no heed

- the log files show nothing much

2 REPLIES 2

Re: Cannot add hosts via the API - heartbeat fails

Super Mentor

@Varun Satokar

If you can share the exact Commands that you used to add host then it will be good to understand what might have gone wrong.

Also please share the ambari-server.log and the ambari-agent.log from the host which you added recently.

1. Have you setup Passwordless from the Ambari Server host to the Newly Added Host?

2. Can you try connecting from Amabri Agent host to ambari Server on the following ports?

# telnet $AMBARI_HOST 8440
# telnet $AMBARI_HOST 8441

.

Re: Cannot add hosts via the API - heartbeat fails

New Contributor

@Jay Kumar SenSharma below is the script being used.

#wait for ambari agent to register with server
    counter=0
    while (( `curl -u admin:admin -i -H 'X-Requested-By: ambari' -X GET ${url}/api/v1/hosts 2> /dev/null | grep ${node} |wc -l` != 2 )) && [ $counter -lt 50 ]; do sleep 10; echo Waiting for ambari agent on node ${node}...; counter=$((counter+1)); done
    
    if (( `curl -u admin:admin -i -H 'X-Requested-By: ambari' -X GET ${url}/api/v1/hosts 2> /dev/null | grep ${node} |wc -l` == 2 )) ; then echo Ambari agent registered with ambari server.; else echo Ambari agent not registered with Ambari server. Exiting.; exit 1; fi
    
     
    
    #POST to add node
    echo Sending request to add node to cluster ${CLUSTER_NAME}...;echo
    if curl -u admin:admin -i -H 'X-Requested-By: ambari' -X POST $url/api/v1/clusters/${CLUSTER_NAME}/hosts/${node}; then echo Request successfully sent...; else echo ERROR: Exiting now; exit 1; fi
    	
    #wait for node addition to cluster
    counter=0
    while ! ( curl -u admin:admin -i -H 'X-Requested-By: ambari' -X GET ${url}/api/v1/clusters/${CLUSTER_NAME}/hosts/${node} 2> /dev/null | grep "\"host_status\" : \"HEALTHY\""  ) && [ $counter -lt 30 ]; do sleep 10; echo Waiting for node ${node} addition...; counter=$((counter+1)); done
	
	if curl -u admin:admin -i -H 'X-Requested-By: ambari' -X GET ${url}/api/v1/clusters/${CLUSTER_NAME}/hosts/${node} 2> /dev/null |grep "\"host_status\" : \"HEALTHY\""|wc -l;  then
    	echo Node ${node} has been added successfully... 
    else
    	echo ERROR: Unable to add node. Please check logs on the agent. Exiting. 
		exit 1
    fi
done


--> Also, the telnet(s) you mentioned were successful.

--> In addition, I noticed something which might help. I introduced a sleep for 100 seconds in the script before it starts to check whether the host is visible in ${url}/api/v1/hosts. And it seems to work well. <br>I don't understand why that should happen though.

Don't have an account?
Coming from Hortonworks? Activate your account here