UPgrading ambari 2.4 to 2.5.0.3 mysqld service is getting STOPPED

[#1]

I am trying to bring back MySQL Server but it not able to remain running, message "This host-level alert is triggered if the Hive Metastore process cannot be determined to be up and listening on the network."

Metastore on g4t8346c.houston.hp.com failed (Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_metastore.py", line 200, in execute
    timeout_kill_strategy=TerminateStrategy.KILL_PROCESS_TREE,
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 262, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 297, in _call
    raise ExecuteTimeoutException(err_msg)
ExecuteTimeoutException: Execution of 'ambari-sudo.sh su ambari-qa -l -s /bin/bash -c 'export  PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/pb/bin:/opt/cmu/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/sbin/:/usr/hdp/current/hive-metastore/bin'"'"' ; export HIVE_CONF_DIR='"'"'/usr/hdp/current/hive-metastore/conf'"'"' ; hive --hiveconf hive.metastore.uris=thrift://g4t8346c.houston.hp.com:9083                 --hiveconf hive.metastore.client.connect.retry.delay=1                 --hiveconf hive.metastore.failure.retries=1                 --hiveconf hive.metastore.connect.retries=1                 --hiveconf hive.metastore.client.socket.timeout=14                 --hiveconf hive.execution.engine=mr -e '"'"'show databases;'"'"''' was killed due timeout after 60 seconds
)

[#2 · Accepted solution]

@Jorge Luis Hernandez Olmos Please check the log file (usually at /var/log/mysql/) and look for the actual error. Also check if there any stale instance of mysqld, is already running. To check this, please execute following command:

ps -ef | grep mysql

If you find any running instance then try to stop mysqld gracefully and if it does not work then kill the daemon. Now try again to start mysql from Ambari.

[#3]

Thank you Manish, this was odd, since logs didn't show anythin in particular, just that session had been successfully terminated, and there was no process running for mysql. After some minutes, we simply try to start the services directly from the linux command line on the server and that bring it up on ambari permanently.

[#4]

@Jorge Luis Hernandez Olmos

I'm happy it worked for you. Sometimes, it happens and I always prefer to take care of mysql from CLI first. As always, if you find this post helpful, don't forget to "accept" answer.