Reply
Highlighted
Explorer
Posts: 10
Registered: ‎11-23-2017

Cloudera Management Services failed to start

[ Edited ]

Hi,

 

We have CDH version 5.15.0 with HA configuration using Corosync and Pacemaker.

We had to stop the cluster - all components, including VMs, were stopped.

When I was restarting the Cloudera Management Services did not start.

We also have sometimes problems that some processes (spark, hive from oozie) do not see a hive database, although the database exists.

 

I'm not able to find in logs any meaningfull informations.

 

I found that in corosync log after restart there are only raws with information about 'lrmd'. There is no raws with 'crmd' and 'pengine' - such raws were before in this logs.


Log before restart:
[root@bdp-lb1 cluster]# more corosync.log-20181106
Nov 05 04:16:55 [2000] bdp-lb1.zzz.com       lrmd:   notice: operation_finished:   cloudera_haproxy_status_60000:10507:stderr [ /etc/init.d/haproxy: line 26: [: =: unary operator expected ]
Nov 05 04:18:35 [2003] bdp-lb1.zzz.com       crmd:     info: crm_timer_popped:     PEngine Recheck Timer (I_PE_CALC) just popped (900000ms)
Nov 05 04:18:35 [2003] bdp-lb1.zzz.com       crmd:   notice: do_state_transition:  State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALCcause=C_TIMER_POPPED origin=crm_timer_popped ]
Nov 05 04:18:35 [2002] bdp-lb1.zzz.com    pengine:     info: process_pe_message:   Input has not changed since last time, not saving to disk
Nov 05 04:18:35 [2002] bdp-lb1.zzz.com    pengine:     info: determine_online_status_fencing:      Node onair-tel-bdp-lb1 is active
...

Log after restart:
[root@lb1 cluster]# more corosync.log
Nov 28 03:28:51 [2208] bdp-lb1.zzz.com       lrmd:   notice: operation_finished:   cloudera_haproxy_status_60000:17261:stderr [ /etc/init.d/haproxy: line 26: [: =: unary operator expected ]
Nov 28 03:29:51 [2208] bdp-lb1.zzz.com       lrmd:   notice: operation_finished:   cloudera_haproxy_status_60000:18015:stderr [ /etc/init.d/haproxy: line 26: [: =: unary operator expected ]
Nov 28 03:30:51 [2208] bdp-lb1.zzz.com       lrmd:   notice: operation_finished:   cloudera_haproxy_status_60000:18786:stderr [ /etc/init.d/haproxy: line 26: [: =: unary operator expected ]
Nov 28 03:31:51 [2208] bdp-lb1.zzz.com       lrmd:   notice: operation_finished:   cloudera_haproxy_status_60000:19541:stderr [ /etc/init.d/haproxy: line 26: [: =: unary operator expected ]
Nov 28 03:32:51 [2208] bdp-lb1.zzz.com       lrmd:   notice: operation_finished:   cloudera_haproxy_status_60000:20293:stderr [ /etc/init.d/haproxy: line 26: [: =: unary operator expected ]
...

 

Thank you

Andrzej

Announcements