07-24-2017 02:33 PM
I had a working CDH5 on RHEL in cluster mode.
My machine hanged somehow,so I had to reboot it.
I see everything in bad shape. I am trying to restart CDH management services, but I see timeout message and no service is starting.
if I start from command line, I see following message:
[root@pc1 Downloads]# service cloudera-scm-agent status ● cloudera-scm-agent.service - LSB: Cloudera SCM Agent Loaded: loaded (/etc/rc.d/init.d/cloudera-scm-agent; bad; vendor preset: disabled) Active: active (exited) since Mon 2017-07-24 15:14:15 CDT; 1h 15min ago Docs: man:systemd-sysv-generator(8) Process: 9482 ExecStop=/etc/rc.d/init.d/cloudera-scm-agent stop (code=exited, status=0/SUCCESS) Process: 9531 ExecStart=/etc/rc.d/init.d/cloudera-scm-agent start (code=exited, status=0/SUCCESS) Jul 24 15:14:14 pc1.localdomain.com systemd: Starting LSB: Cloudera SCM A.... Jul 24 15:14:14 pc1.localdomain.com su: (to root) root on none Jul 24 15:14:15 pc1.localdomain.com cloudera-scm-agent: Starting cloude... Jul 24 15:14:15 pc1.localdomain.com systemd: Started LSB: Cloudera SCM Agent.
I am not sure what has gone wrong.
07-24-2017 03:19 PM
There could be many reasons
1. CM -> hosts -> check the hosts are in green
2. Login as root and start ntpd service if it is not started automatically
service ntpd status
service ntpd start
chkconfig --list ntpd
chkconfig ntpd on;
wait for few mins and check the hosts status again.
post more logs if you are still facing issue
07-24-2017 03:50 PM
following are the logs with ntpd:
[root@pc1 Downloads]# service ntpd status Redirecting to /bin/systemctl status ntpd.service ● ntpd.service - Network Time Service Loaded: loaded (/usr/lib/systemd/system/ntpd.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2017-07-24 17:40:31 CDT; 1min 8s ago Process: 16053 ExecStart=/usr/sbin/ntpd -u ntp:ntp $OPTIONS (code=exited, status=0/SUCCESS) Main PID: 16054 (ntpd) CGroup: /system.slice/ntpd.service └─16054 /usr/sbin/ntpd -u ntp:ntp -g Jul 24 17:40:31 pc1.localdomain.com ntpd: Listen normally on 3 wlp3s0 192.168.1.115 UDP 123 [root@pc1 Downloads]# chkconfig --list ntpd error reading information on service ntpd: No such file or directory [root@pc1 Downloads]# chkconfig ntpd on Note: Forwarding request to 'systemctl enable ntpd.service'.
I am in a fix now where to start now.
07-24-2017 06:51 PM - edited 07-24-2017 06:55 PM
Could you let me know if you are able to bring up your Cloudera-scm-server / Cloudera-scm-server-db ?
whats their status ?
where you able to bring up the Cloudera manager web ui ?
could you check if you multiple java version hanging around in the host ?
or you use alternatives
Meantime using the Code tag tool bar could you share the logs cloudera-scm-server / cloudera-scm-agent ?
is the location for those logs
07-24-2017 08:05 PM
It is very wierd that I rebooted the system, started the cloudera-management-services, and then started others services individually. It is working now.
I am not sure why it happened. It was really a nightmare, and I am really surprised what should I do in such scenarios (without rebooting system).