Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

cloudera not starting after rebooting machine

SOLVED Go to solution

cloudera not starting after rebooting machine

Explorer

I had a working CDH5  on RHEL in cluster mode.

 

My machine hanged somehow,so I had to reboot it.

 

I see everything in bad shape. I am trying to restart CDH management services, but I see timeout message and no service is starting.

 

if I start from command line, I see following message:

 

[root@pc1 Downloads]# service cloudera-scm-agent  status 
● cloudera-scm-agent.service - LSB: Cloudera SCM Agent
   Loaded: loaded (/etc/rc.d/init.d/cloudera-scm-agent; bad; vendor preset: disabled)
   Active: active (exited) since Mon 2017-07-24 15:14:15 CDT; 1h 15min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 9482 ExecStop=/etc/rc.d/init.d/cloudera-scm-agent stop (code=exited, status=0/SUCCESS)
  Process: 9531 ExecStart=/etc/rc.d/init.d/cloudera-scm-agent start (code=exited, status=0/SUCCESS)

Jul 24 15:14:14 pc1.localdomain.com systemd[1]: Starting LSB: Cloudera SCM A....
Jul 24 15:14:14 pc1.localdomain.com su[9545]: (to root) root on none
Jul 24 15:14:15 pc1.localdomain.com cloudera-scm-agent[9531]: Starting cloude...
Jul 24 15:14:15 pc1.localdomain.com systemd[1]: Started LSB: Cloudera SCM Agent.

I am not sure what has gone wrong.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: cloudera not starting after rebooting machine

Explorer

It is very wierd that I rebooted the system, started the cloudera-management-services, and then started others services individually. It is working now.

 

I am not sure why it happened. It was really a nightmare, and I am really surprised what should I do in such scenarios (without rebooting system).

4 REPLIES 4

Re: cloudera not starting after rebooting machine

Champion

@cllearner

 

There could be many reasons

 

1. CM -> hosts -> check the hosts are in green

2. Login as root and start ntpd service if it is not started automatically

 

service ntpd status

service ntpd start

chkconfig  --list ntpd

chkconfig ntpd on;

 

wait for few mins and check the hosts status again. 

 

post more logs if you are still facing issue

Highlighted

Re: cloudera not starting after rebooting machine

Explorer

following are the logs with ntpd:

 

[root@pc1 Downloads]# service ntpd status
Redirecting to /bin/systemctl status  ntpd.service
● ntpd.service - Network Time Service
   Loaded: loaded (/usr/lib/systemd/system/ntpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2017-07-24 17:40:31 CDT; 1min 8s ago
  Process: 16053 ExecStart=/usr/sbin/ntpd -u ntp:ntp $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 16054 (ntpd)
   CGroup: /system.slice/ntpd.service
           └─16054 /usr/sbin/ntpd -u ntp:ntp -g

Jul 24 17:40:31 pc1.localdomain.com ntpd[16054]: Listen normally on 3 wlp3s0 192.168.1.115 UDP 123

[root@pc1 Downloads]# chkconfig --list ntpd
error reading information on service ntpd: No such file or directory
[root@pc1 Downloads]# chkconfig ntpd on
Note: Forwarding request to 'systemctl enable ntpd.service'.

Screenshot from 2017-07-24 17-46-06.pngScreenshot from 2017-07-24 17-44-51.png

 

I am in a fix now where to start now.

Re: cloudera not starting after rebooting machine

Champion

@cllearner

 

Could you let me know if you are able to bring up your Cloudera-scm-server / Cloudera-scm-server-db ? 

whats their status ? 

 

where you able to bring up the Cloudera manager web ui ?

 

could  you check if you multiple java version hanging around in the host  ? 

/usr/java/ 

/usr/lib/jvm/

or you use alternatives

 

Meantime using the Code tag tool bar could you share the logs cloudera-scm-server / cloudera-scm-agent ?

is the location for those logs 

 

/var/log/

 

 

 

Re: cloudera not starting after rebooting machine

Explorer

It is very wierd that I rebooted the system, started the cloudera-management-services, and then started others services individually. It is working now.

 

I am not sure why it happened. It was really a nightmare, and I am really surprised what should I do in such scenarios (without rebooting system).