Hi,
I work with cdh5.4.7 and I have the same issue, and I resolved it.
When Cloudera Manager server is restarted after upgrade or maintance tasks this starts cloudera server and cloudera agent, but it does not start Cloudera Management Services (mgmt).
The reason is because cloudera-scm-server and cloudera-scm-agent is configured to start at the same time:
[ cloudera_server ]: grep chkconfig /etc/init.d/cloudera-scm-*
/etc/init.d/cloudera-scm-agent:# chkconfig: 2345 90 10
/etc/init.d/cloudera-scm-server:# chkconfig: 2345 90 10
Cloudera Agent start Cloudera Management Services (mgmt) and it needs to connect to Cloudera Server, Cloudera Server takes more time to start than Cloudera Agent. Cloudera Agent tries to start mgmt 5 times with only 2 seconds between every retry, finally mgmt cannot start (in mgmt role logs I can see "connection refused" errors):
2017-01-02 15:06:44,673 WARN com.cloudera.cmon.firehose.Main: No descriptor fetched from https://cloudera_server:7183 on after 1 tries, sleeping...
2017-01-02 15:06:44,798 WARN com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: Failed to publish event: SimpleEvent{attributes={ROLE_TYPE=[SERVICEMONITOR], EXCEPTION_TYPES=[java.net.ConnectException], HOST_IDS=[..], STACKTRACE=[java.net.ConnectException: Connection refused
[..]
2017-01-02 15:06:46,708 WARN com.cloudera.cmon.firehose.Main: No descriptor fetched from https://cloudera_server:7183 on after 2 tries, sleeping...
[..]
2017-01-02 15:06:52,724 WARN com.cloudera.cmon.firehose.Main: No descriptor fetched from https://cloudera_server:7183 on after 5 tries, sleeping...
To temporally fix this issue I do that:
1. Change start order:
I changed to this (server 90 to 89):
[ cloudera_server ]: grep chkconfig /etc/init.d/cloudera-scm-*
/etc/init.d/cloudera-scm-agent:# chkconfig: 2345 90 10
/etc/init.d/cloudera-scm-server:# chkconfig: 2345 89 10
2. Add cloudera server check in agent init start script.
/etc/init.d/cloudera-scm-agent (green line):
---
[..]
start() {
[..]
+ for i in $(seq 1 30); do curl -k -s -I $(facter cdh_url | awk -F\/api '{print $1}') | grep -q '200 OK' &>/tmp/init_cloudera_agent.out && break; sleep 10; done
$CMF_SUDO_CMD /bin/bash -c "nohup $AGENT_SCRIPT $CMF_AGENT_ARGS" >> $AGENT_OUT 2>&1 </dev/null &
[..]
}
[..]
---
* cdh_url is a custom facter that retur https://cloudera_server:7183/api/v10
If I only change start order it doesn't works because when "/etc/init.d/cloudera-scm-server start" is executed it doesn't wait to be completely started, it returns OK immediately (but is starting yet in background). When I reboot this server it starts cloudera-scm-server and immediately starts cloudera-scm-agent, cloudera-scm-agent starts faster than cloudera-scm-server and mgmt cannot connect to cloudera server web, after 5 tries it still down and I need to start mgmt manually.
If I do this changes it works fine, but I think that I should not change this configurations…
Another valid solution is that cloudera-scm-server waits to be successful and completely started to return OK and start first server and then agent with mgmt services, but for the moment it works for me.
Marc.
