Hi,
We have CDH version 5.15.0 with HA configuration using Corosync and Pacemaker.
We had to stop the cluster - all components, including VMs, were stopped.
When I was restarting the Cloudera Management Services did not start.
We also have sometimes problems that some processes (spark, hive from oozie) do not see a hive database, although the database exists.
I'm not able to find in logs any meaningfull informations.
I found that in corosync log after restart there are only raws with information about 'lrmd'. There is no raws with 'crmd' and 'pengine' - such raws were before in this logs.
Log before restart:
[root@bdp-lb1 cluster]# more corosync.log-20181106
Nov 05 04:16:55 [2000] bdp-lb1.zzz.com lrmd: notice: operation_finished: cloudera_haproxy_status_60000:10507:stderr [ /etc/init.d/haproxy: line 26: [: =: unary operator expected ]
Nov 05 04:18:35 [2003] bdp-lb1.zzz.com crmd: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped (900000ms)
Nov 05 04:18:35 [2003] bdp-lb1.zzz.com crmd: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALCcause=C_TIMER_POPPED origin=crm_timer_popped ]
Nov 05 04:18:35 [2002] bdp-lb1.zzz.com pengine: info: process_pe_message: Input has not changed since last time, not saving to disk
Nov 05 04:18:35 [2002] bdp-lb1.zzz.com pengine: info: determine_online_status_fencing: Node onair-tel-bdp-lb1 is active
...
Log after restart:
[root@lb1 cluster]# more corosync.log
Nov 28 03:28:51 [2208] bdp-lb1.zzz.com lrmd: notice: operation_finished: cloudera_haproxy_status_60000:17261:stderr [ /etc/init.d/haproxy: line 26: [: =: unary operator expected ]
Nov 28 03:29:51 [2208] bdp-lb1.zzz.com lrmd: notice: operation_finished: cloudera_haproxy_status_60000:18015:stderr [ /etc/init.d/haproxy: line 26: [: =: unary operator expected ]
Nov 28 03:30:51 [2208] bdp-lb1.zzz.com lrmd: notice: operation_finished: cloudera_haproxy_status_60000:18786:stderr [ /etc/init.d/haproxy: line 26: [: =: unary operator expected ]
Nov 28 03:31:51 [2208] bdp-lb1.zzz.com lrmd: notice: operation_finished: cloudera_haproxy_status_60000:19541:stderr [ /etc/init.d/haproxy: line 26: [: =: unary operator expected ]
Nov 28 03:32:51 [2208] bdp-lb1.zzz.com lrmd: notice: operation_finished: cloudera_haproxy_status_60000:20293:stderr [ /etc/init.d/haproxy: line 26: [: =: unary operator expected ]
...
Thank you
Andrzej
