Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

cloudera not starting after rebooting machine

avatar
Explorer

I had a working CDH5  on RHEL in cluster mode.

 

My machine hanged somehow,so I had to reboot it.

 

I see everything in bad shape. I am trying to restart CDH management services, but I see timeout message and no service is starting.

 

if I start from command line, I see following message:

 

[root@pc1 Downloads]# service cloudera-scm-agent  status 
● cloudera-scm-agent.service - LSB: Cloudera SCM Agent
   Loaded: loaded (/etc/rc.d/init.d/cloudera-scm-agent; bad; vendor preset: disabled)
   Active: active (exited) since Mon 2017-07-24 15:14:15 CDT; 1h 15min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 9482 ExecStop=/etc/rc.d/init.d/cloudera-scm-agent stop (code=exited, status=0/SUCCESS)
  Process: 9531 ExecStart=/etc/rc.d/init.d/cloudera-scm-agent start (code=exited, status=0/SUCCESS)

Jul 24 15:14:14 pc1.localdomain.com systemd[1]: Starting LSB: Cloudera SCM A....
Jul 24 15:14:14 pc1.localdomain.com su[9545]: (to root) root on none
Jul 24 15:14:15 pc1.localdomain.com cloudera-scm-agent[9531]: Starting cloude...
Jul 24 15:14:15 pc1.localdomain.com systemd[1]: Started LSB: Cloudera SCM Agent.

I am not sure what has gone wrong.

1 ACCEPTED SOLUTION

avatar
Explorer

It is very wierd that I rebooted the system, started the cloudera-management-services, and then started others services individually. It is working now.

 

I am not sure why it happened. It was really a nightmare, and I am really surprised what should I do in such scenarios (without rebooting system).

View solution in original post

4 REPLIES 4

avatar
Champion

@cllearner

 

There could be many reasons

 

1. CM -> hosts -> check the hosts are in green

2. Login as root and start ntpd service if it is not started automatically

 

service ntpd status

service ntpd start

chkconfig  --list ntpd

chkconfig ntpd on;

 

wait for few mins and check the hosts status again. 

 

post more logs if you are still facing issue

avatar
Explorer

following are the logs with ntpd:

 

[root@pc1 Downloads]# service ntpd status
Redirecting to /bin/systemctl status  ntpd.service
● ntpd.service - Network Time Service
   Loaded: loaded (/usr/lib/systemd/system/ntpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2017-07-24 17:40:31 CDT; 1min 8s ago
  Process: 16053 ExecStart=/usr/sbin/ntpd -u ntp:ntp $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 16054 (ntpd)
   CGroup: /system.slice/ntpd.service
           └─16054 /usr/sbin/ntpd -u ntp:ntp -g

Jul 24 17:40:31 pc1.localdomain.com ntpd[16054]: Listen normally on 3 wlp3s0 192.168.1.115 UDP 123

[root@pc1 Downloads]# chkconfig --list ntpd
error reading information on service ntpd: No such file or directory
[root@pc1 Downloads]# chkconfig ntpd on
Note: Forwarding request to 'systemctl enable ntpd.service'.

Screenshot from 2017-07-24 17-46-06.pngScreenshot from 2017-07-24 17-44-51.png

 

I am in a fix now where to start now.

avatar
Champion

@cllearner

 

Could you let me know if you are able to bring up your Cloudera-scm-server / Cloudera-scm-server-db ? 

whats their status ? 

 

where you able to bring up the Cloudera manager web ui ?

 

could  you check if you multiple java version hanging around in the host  ? 

/usr/java/ 

/usr/lib/jvm/

or you use alternatives

 

Meantime using the Code tag tool bar could you share the logs cloudera-scm-server / cloudera-scm-agent ?

is the location for those logs 

 

/var/log/

 

 

 

avatar
Explorer

It is very wierd that I rebooted the system, started the cloudera-management-services, and then started others services individually. It is working now.

 

I am not sure why it happened. It was really a nightmare, and I am really surprised what should I do in such scenarios (without rebooting system).