Reply
Explorer
Posts: 17
Registered: ‎06-03-2017
Accepted Solution

cloudera not starting after rebooting machine

I had a working CDH5  on RHEL in cluster mode.

 

My machine hanged somehow,so I had to reboot it.

 

I see everything in bad shape. I am trying to restart CDH management services, but I see timeout message and no service is starting.

 

if I start from command line, I see following message:

 

[root@pc1 Downloads]# service cloudera-scm-agent  status 
● cloudera-scm-agent.service - LSB: Cloudera SCM Agent
   Loaded: loaded (/etc/rc.d/init.d/cloudera-scm-agent; bad; vendor preset: disabled)
   Active: active (exited) since Mon 2017-07-24 15:14:15 CDT; 1h 15min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 9482 ExecStop=/etc/rc.d/init.d/cloudera-scm-agent stop (code=exited, status=0/SUCCESS)
  Process: 9531 ExecStart=/etc/rc.d/init.d/cloudera-scm-agent start (code=exited, status=0/SUCCESS)

Jul 24 15:14:14 pc1.localdomain.com systemd[1]: Starting LSB: Cloudera SCM A....
Jul 24 15:14:14 pc1.localdomain.com su[9545]: (to root) root on none
Jul 24 15:14:15 pc1.localdomain.com cloudera-scm-agent[9531]: Starting cloude...
Jul 24 15:14:15 pc1.localdomain.com systemd[1]: Started LSB: Cloudera SCM Agent.

I am not sure what has gone wrong.

Posts: 519
Topics: 14
Kudos: 92
Solutions: 45
Registered: ‎09-02-2016

Re: cloudera not starting after rebooting machine

@cllearner

 

There could be many reasons

 

1. CM -> hosts -> check the hosts are in green

2. Login as root and start ntpd service if it is not started automatically

 

service ntpd status

service ntpd start

chkconfig  --list ntpd

chkconfig ntpd on;

 

wait for few mins and check the hosts status again. 

 

post more logs if you are still facing issue

Explorer
Posts: 17
Registered: ‎06-03-2017

Re: cloudera not starting after rebooting machine

following are the logs with ntpd:

 

[root@pc1 Downloads]# service ntpd status
Redirecting to /bin/systemctl status  ntpd.service
● ntpd.service - Network Time Service
   Loaded: loaded (/usr/lib/systemd/system/ntpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2017-07-24 17:40:31 CDT; 1min 8s ago
  Process: 16053 ExecStart=/usr/sbin/ntpd -u ntp:ntp $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 16054 (ntpd)
   CGroup: /system.slice/ntpd.service
           └─16054 /usr/sbin/ntpd -u ntp:ntp -g

Jul 24 17:40:31 pc1.localdomain.com ntpd[16054]: Listen normally on 3 wlp3s0 192.168.1.115 UDP 123

[root@pc1 Downloads]# chkconfig --list ntpd
error reading information on service ntpd: No such file or directory
[root@pc1 Downloads]# chkconfig ntpd on
Note: Forwarding request to 'systemctl enable ntpd.service'.

Screenshot from 2017-07-24 17-46-06.pngScreenshot from 2017-07-24 17-44-51.png

 

I am in a fix now where to start now.

Champion
Posts: 777
Registered: ‎05-16-2016

Re: cloudera not starting after rebooting machine

[ Edited ]

@cllearner

 

Could you let me know if you are able to bring up your Cloudera-scm-server / Cloudera-scm-server-db ? 

whats their status ? 

 

where you able to bring up the Cloudera manager web ui ?

 

could  you check if you multiple java version hanging around in the host  ? 

/usr/java/ 

/usr/lib/jvm/

or you use alternatives

 

Meantime using the Code tag tool bar could you share the logs cloudera-scm-server / cloudera-scm-agent ?

is the location for those logs 

 

/var/log/

 

 

 

Explorer
Posts: 17
Registered: ‎06-03-2017

Re: cloudera not starting after rebooting machine

It is very wierd that I rebooted the system, started the cloudera-management-services, and then started others services individually. It is working now.

 

I am not sure why it happened. It was really a nightmare, and I am really surprised what should I do in such scenarios (without rebooting system).