i have installed all services on one node and had few issue: after finishing the install stage Hive Metastore fail to start with same issue as before: INFO 2017-10-17 01:55:04,537 RecoveryManager.py:255 - SPARK2_THRIFTSERVER needs recovery, desired = STARTED, and current = INSTALLED.
INFO 2017-10-17 01:55:04,537 RecoveryManager.py:255 - SPARK_THRIFTSERVER needs recovery, desired = STARTED, and current = INSTALLED.
INFO 2017-10-17 01:55:04,538 RecoveryManager.py:255 - HIVE_METASTORE needs recovery, desired = STARTED, and current = INSTALLED.
INFO 2017-10-17 01:55:09,219 ClusterConfiguration.py:119 - Updating cached configurations for cluster vqcluster
INFO 2017-10-17 01:55:09,252 RecoveryManager.py:717 - Received EXECUTION_COMMAND (START), desired state of HIVE_METASTORE to STARTED
INFO 2017-10-17 01:55:09,253 Controller.py:249 - Adding 1 commands. Heartbeat id = 62015
INFO 2017-10-17 01:55:09,253 ActionQueue.py:113 - Adding EXECUTION_COMMAND for role HIVE_METASTORE for service HIVE of cluster vqcluster to the queue.
INFO 2017-10-17 01:55:09,288 ActionQueue.py:238 - Executing command with id = 43-0, taskId = 276 for role = HIVE_METASTORE of cluster vqcluster.
INFO 2017-10-17 01:55:09,288 ActionQueue.py:279 - Command execution metadata - taskId = 276, retry enabled = False, max retry duration (sec) = 0, log_output = True
INFO 2017-10-17 01:55:09,289 CustomServiceOrchestrator.py:265 - Generating the JCEKS file: roleCommand=START and taskId = 276
INFO 2017-10-17 01:55:09,289 CustomServiceOrchestrator.py:243 - Identifying config hive-site for CS:
INFO 2017-10-17 01:55:09,289 CustomServiceOrchestrator.py:288 - provider_path=jceks://file/var/lib/ambari-agent/cred/conf/hive/hive-site.jceks
INFO 2017-10-17 01:55:09,289 CustomServiceOrchestrator.py:295 - ('/usr/jdk64/jdk1.8.0_112/bin/java', '-cp', '/var/lib/ambari-agent/cred/lib/*', 'org.apache.hadoop.security.alias.CredentialShell', 'create', u'javax.jdo.option.ConnectionPassword', '-value', [PROTECTED], '-provider', 'jceks://file/var/lib/ambari-agent/cred/conf/hive/hive-site.jceks')
WARNING 2017-10-17 01:55:09,318 CommandStatusDict.py:128 - [Errno 2] No such file or directory: '/var/lib/ambari-agent/data/output-276.txt'
INFO 2017-10-17 01:55:09,694 CustomServiceOrchestrator.py:297 - cmd_result = 0
INFO 2017-10-17 01:55:14,801 RecoveryManager.py:255 - SPARK2_THRIFTSERVER needs recovery, desired = STARTED, and current = INSTALLED.
INFO 2017-10-17 01:55:14,801 RecoveryManager.py:255 - SPARK_THRIFTSERVER needs recovery, desired = STARTED, and current = INSTALLED.
INFO 2017-10-17 01:55:14,801 RecoveryManager.py:255 - HIVE_METASTORE needs recovery, desired = STARTED, and current = INSTALLED.
INFO 2017-10-17 01:55:15,375 PythonExecutor.py:130 - Command ['/usr/bin/python',
u'/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py',
u'START',
'/var/lib/ambari-agent/data/command-276.json',
u'/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package',
'/var/lib/ambari-agent/data/structured-out-276.json',
'INFO',
'/var/lib/ambari-agent/tmp',
'PROTOCOL_TLSv1',
''] failed with exitcode=1 and Spark2 Thrift Server, Spark(1) Thrift Server can start but keep going down, ERROR 2017-10-17 01:57:36,029 script_alert.py:123 - [Alert][spark_thriftserver_status] Failed with result CRITICAL: ['Connection failed on host ambari-master.test.com:10015 (Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/alerts/alert_spark_thrift_port.py", line 143, in execute
Execute(cmd, user=hiveruser, path=[beeline_cmd], timeout=CHECK_COMMAND_TIMEOUT_DEFAULT)
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 166, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 262, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
raise ExecutionFailed(err_msg, code, out, err)
ExecutionFailed: Execution of \'! beeline -u \'jdbc:hive2://ambari-master.test.com:10015/default\' transportMode=binary -e \'\' 2>&1| awk \'{print}\'|grep -i -e \'Connection refused\' -e \'Invalid URL\'\' returned 1. Error: Could not open client transport with JDBC Uri: jdbc:hive2://ambari-master.test.com:10015/default: java.net.ConnectException: Connection refused (Connection refused) (state=08S01,code=0)
Error: Could not open client transport with JDBC Uri: jdbc:hive2://ambari-master.test.com:10015/default: java.net.ConnectException: Connection refused (Connection refused) (state=08S01,code=0)
)']
our main requirement is spark2 so ill try to make another clean install with only spark2 and its dependencies hoping not to have any more issues
... View more
