Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Services won't restart after firmware upgrade

Services won't restart after firmware upgrade

Explorer

I have an issue with all of our services, we did a firmware upgrade for our machine and now all of the services say:

 

"No host heartbeat; CDH versions cannot be verified."

 

and I cannot restart the services through the manager they will either Time-out or mention cannot communicate with the name node:

 

"Command aborted because of exception: Command timed-out after 150 seconds"

here is the log for our cluster, just a heads up the cluster is one machine and the data nodes are vm's. 

 

The log file is too big to post please observe here:
http://pastebin.com/CznqykHF

 

 

7 REPLIES 7
Highlighted

Re: Services won't restart after firmware upgrade

Expert Contributor

 My first steps for troubleshooting something like this would be to test network connectivity. 

 

was there a reboot associated with the firmware update? Maybe some firewall rules reverted back post reboot? 

Re: Services won't restart after firmware upgrade

Explorer

Sorry, so my steps included running ifconfig on each node and testing to see if they 

  1. Ping each other (They can)
  2. Are still the same pre-reboot (they are)

 

Then I checked the hosts file to ensure each node was resolving to the FQDN and ip address

 

regarding the firewalls 

 

  1. I checked SELinux and it was off on all nodes
  2. I checked IPTables and again they were still off.

 

What recommendations would you take after these steps?

Re: Services won't restart after firmware upgrade

Expert Contributor

I think the only other thing i would test is to use something that can connect over TCP to verify none of the ports are being blocked by something other than IPTables (though i'm not sure what that would be)

 

in rhel for example:

 

nc -z <host> <port> 

 

ping uses ICMP which is controlled in different ways from TCP. 

 

Re: Services won't restart after firmware upgrade

Expert Contributor

If it's not network, looks like you are using cloudera manager? if so I'd defer to any cloudera employee as they have access to the code to better determine what could cause your issues. 

Re: Services won't restart after firmware upgrade

If network connectivity is working

- please look up the Hosts tab in Cloudera Manager. Do you see all your slave nodes listed correctly?

- Are you using TLS between the nodes and CM?

- What do the agent logs under /var/log/cloudera-scm-agent/ say?

- Check /etc/cloudera-scm-agent/config.ini for the CM host they're meant to be heartbeat to.

 

Regards
Gautam

Regards,
Gautam Gopalakrishnan

Re: Services won't restart after firmware upgrade

Explorer
I re posted this inside here because we did use the cloudera binary to install the services it answers your questions

http://community.cloudera.com/t5/Cloudera-Manager-Installation/After-hard-restart-cant-start-any-ser...

Re: Services won't restart after firmware upgrade

Since you are using cloudera manager we should abandon this thread and continue in the other one

http://community.cloudera.com/t5/Cloudera-Manager-Installation/After-hard-restart-cant-start-any-ser...

Regards,
Gautam Gopalakrishnan
Don't have an account?
Coming from Hortonworks? Activate your account here