I work on a small dev cluster (4 machines) running on HDP 3.0.1 and recently lost all heartbeat on the master node due to an unplanned upgrade of my OS (ubuntu16 to 18). I still don't know why or how this node got its OS upgraded but I can't rollback to the previous version.
The log from ambari-server reads:
2019-05-20 17:06:50,553 INFO [agent-register-processor-7] HeartBeatHandler:317 - agentOsType = ubuntu18 2019-05-20 17:06:50,554 WARN [agent-register-processor-7] HeartBeatHandler:319 - Received registration request from host with not supported os type, hostname=hadoopmaster.*****.***, serverOsType=ubuntu16, agentOsType=ubuntu18 2019-05-20 17:06:50,554 INFO [agent-register-processor-7] HeartbeatController:105 - Cannot register host with not supported os type, hostname=hadoopmaster.*****.***, serverOsType=ubuntu16, agentOsType=ubuntu18 org.apache.ambari.server.AmbariException: Cannot register host with not supported os type, hostname=hadoopmaster.*****.***, serverOsType=ubuntu16, agentOsType=ubuntu18 at org.apache.ambari.server.agent.HeartBeatHandler.handleRegistration(HeartBeatHandler.java:327) at org.apache.ambari.server.agent.stomp.HeartbeatController.lambda$register$0(HeartbeatController.java:100) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
I stopped all services using ambari interface and tried to use the "Recover host" option but I get the following message:
I'm not sure how to interpret this error. Are the services all stopped but ambari is unable to get that information or are there some services still running on the master node that I need to turn off manually? The master node has been rebooted since the OS upgrade and there is no script to automatically start ambari services so they should all be stopped.Would a host recovery even work in the case of an OS upgrade?
I read that the last ambari version supports Ubuntu18. Would an upgrade fix the problem?
Thank you for your help and advices.
Please make sure that your Ambari Server and Agent versions are atleast 2.7.3 if you want to use "ubuntu-18".
Please refer to the following link and then click on the ambari version tat you are using to findout if the OS that you are using is supported or not: https://supportmatrix.hortonworks.com/
Thank you for your answer. The OS upgrade (Ubuntu16 - Ubuntu18) wasn't planned and so my current ambari version is 2.7.0 which does not support Ubuntu18.
Do you think it is possible to upgrade ambari 2.7.3 and solve the heartbeat problem or am I forced to reinstall the whole cluster?