Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Components won't start

Components won't start

Explorer

Hi All,

 

I have an Ambari server running with HDP 3.1 in SLES 12sp5. 

Hive MetaSote, HST server, Metrics collector, grafana, namenode, resource manager, snamenode, yarn registry dns zookeeper, data node, hst agent, metrics monitor, nodemanager, hdfs client, hive client, mapreduce2 client, spark2 client, tex client, yarn client and zookeeper client all start and work fine. 

 

But i can't get History Server/Mapreduce2, HiveServer2, Spark2History Server and the Timeline Serivice c2.0 reader to start. 

 

All the logs look the same with this error:

 

Traceback (most recent call last):
File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 287, in _run_command
result_dict = json.loads(out)
File "/usr/lib/ambari-agent/lib/ambari_simplejson/__init__.py", line 307, in loads
return _default_decoder.decode(s)
File "/usr/lib/ambari-agent/lib/ambari_simplejson/decoder.py", line 335, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/ambari-agent/lib/ambari_simplejson/decoder.py", line 353, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

The above exception was the cause of the following exception:

Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_server.py", line 143, in <module>
HiveServer().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
method(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_server.py", line 50, in start
self.configure(env) # FOR SECURITY
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_server.py", line 45, in configure
hive(name='hiveserver2')
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive.py", line 119, in hive
setup_hiveserver2()
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive.py", line 149, in setup_hiveserver2
copy_to_hdfs("mapreduce", params.user_group, params.hdfs_user, skip=params.sysprep_skip_copy_tarballs_hdfs)
File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/copy_tarball.py", line 516, in copy_to_hdfs
replace_existing_files=replace_existing_files,
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 677, in action_create_on_execute
self.action_delayed("create")
File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 674, in action_delayed
self.get_hdfs_resource_executor().action_delayed(action_name, self)
File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 373, in action_delayed
self.action_delayed_for_nameservice(None, action_name, main_resource)
File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 403, in action_delayed_for_nameservice
self._create_resource()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 419, in _create_resource
self._create_file(self.main_resource.resource.target, source=self.main_resource.resource.source, mode=self.mode)
File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 534, in _create_file
self.util.run_command(target, 'CREATE', method='PUT', overwrite=True, assertable_result=False, file_to_put=source, **kwargs)
File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 214, in run_command
return self._run_command(*args, **kwargs)
File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 295, in _run_command
raise WebHDFSCallException(err_msg, result_dict)
resource_management.libraries.providers.hdfs_resource.WebHDFSCallException: Execution of 'curl -sS -L -w '%{http_code}' -X PUT --data-binary @/usr/hdp/3.1.0.0-78/hadoop/mapreduce.tar.gz -H 'Content-Type: application/octet-stream' 'http://(SERVER).domain:50070/webhdfs/v1/hdp/apps/3.1.0.0-78/mapreduce/mapreduce.tar.gz?op=CREATE&user.name=hdfs&overwrite=True&permission=444'' returned status_code=502.
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Bad Gateway</title>
</head><body>
<h1>Bad Gateway</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
</p>
</body></html>
stdout: /var/lib/ambari-agent/data/output-322.txt

2020-05-04 21:18:34,885 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.0.0-78 -> 3.1.0.0-78
2020-05-04 21:18:34,903 - Using hadoop conf dir: /usr/hdp/3.1.0.0-78/hadoop/conf
2020-05-04 21:18:35,114 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.0.0-78 -> 3.1.0.0-78
2020-05-04 21:18:35,118 - Using hadoop conf dir: /usr/hdp/3.1.0.0-78/hadoop/conf
2020-05-04 21:18:35,119 - Group['livy'] {}
2020-05-04 21:18:35,120 - Group['spark'] {}
2020-05-04 21:18:35,121 - Group['hdfs'] {}
2020-05-04 21:18:35,121 - Group['hadoop'] {}
2020-05-04 21:18:35,121 - Group['users'] {}
2020-05-04 21:18:35,121 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2020-05-04 21:18:35,122 - User['yarn-ats'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2020-05-04 21:18:35,123 - User['livy'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['livy', 'hadoop'], 'uid': None}
2020-05-04 21:18:35,123 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2020-05-04 21:18:35,124 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['spark', 'hadoop'], 'uid': None}
2020-05-04 21:18:35,125 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2020-05-04 21:18:35,125 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}
2020-05-04 21:18:35,126 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}
2020-05-04 21:18:35,126 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hadoop'], 'uid': None}
2020-05-04 21:18:35,127 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2020-05-04 21:18:35,128 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2020-05-04 21:18:35,128 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2020-05-04 21:18:35,130 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2020-05-04 21:18:35,136 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if
2020-05-04 21:18:35,137 - Group['hdfs'] {}
2020-05-04 21:18:35,137 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hadoop', u'hdfs']}
2020-05-04 21:18:35,137 - FS Type: HDFS
2020-05-04 21:18:35,137 - Directory['/etc/hadoop'] {'mode': 0755}
2020-05-04 21:18:35,150 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2020-05-04 21:18:35,151 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2020-05-04 21:18:35,173 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2020-05-04 21:18:35,180 - Skipping Execute[('setenforce', '0')] due to not_if
2020-05-04 21:18:35,181 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
2020-05-04 21:18:35,183 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
2020-05-04 21:18:35,183 - Directory['/var/run/hadoop/hdfs'] {'owner': 'hdfs', 'cd_access': 'a'}
2020-05-04 21:18:35,184 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}
2020-05-04 21:18:35,187 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2020-05-04 21:18:35,189 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2020-05-04 21:18:35,194 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2020-05-04 21:18:35,202 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/hadoop-metrics2.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2020-05-04 21:18:35,203 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2020-05-04 21:18:35,204 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2020-05-04 21:18:35,207 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop', 'mode': 0644}
2020-05-04 21:18:35,212 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2020-05-04 21:18:35,217 - Skipping unlimited key JCE policy check and setup since the Java VM is not managed by Ambari
2020-05-04 21:18:35,566 - Using hadoop conf dir: /usr/hdp/3.1.0.0-78/hadoop/conf
2020-05-04 21:18:35,575 - call['ambari-python-wrap /usr/bin/hdp-select status hive-server2'] {'timeout': 20}
2020-05-04 21:18:35,602 - call returned (0, 'hive-server2 - 3.1.0.0-78')
2020-05-04 21:18:35,604 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.0.0-78 -> 3.1.0.0-78
2020-05-04 21:18:35,627 - File['/var/lib/ambari-agent/cred/lib/CredentialUtil.jar'] {'content': DownloadSource('http://(SERVER):8080/resources/CredentialUtil.jar'), 'mode': 0755}
2020-05-04 21:18:35,628 - Not downloading the file from http://(SERVER):8080/resources/CredentialUtil.jar, because /var/lib/ambari-agent/tmp/CredentialUtil.jar already exists
2020-05-04 21:18:36,412 - Directories to fill with configs: [u'/usr/hdp/current/hive-server2/conf', u'/usr/hdp/current/hive-server2/conf/']
2020-05-04 21:18:36,412 - Directory['/etc/hive/3.1.0.0-78/0'] {'owner': 'hive', 'group': 'hadoop', 'create_parents': True, 'mode': 0755}
2020-05-04 21:18:36,413 - XmlConfig['mapred-site.xml'] {'group': 'hadoop', 'conf_dir': '/etc/hive/3.1.0.0-78/0', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'hive', 'configurations': ...}
2020-05-04 21:18:36,422 - Generating config: /etc/hive/3.1.0.0-78/0/mapred-site.xml
2020-05-04 21:18:36,422 - File['/etc/hive/3.1.0.0-78/0/mapred-site.xml'] {'owner': 'hive', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2020-05-04 21:18:36,455 - Writing File['/etc/hive/3.1.0.0-78/0/mapred-site.xml'] because contents don't match
2020-05-04 21:18:36,456 - File['/etc/hive/3.1.0.0-78/0/hive-default.xml.template'] {'owner': 'hive', 'group': 'hadoop', 'mode': 0644}
2020-05-04 21:18:36,456 - File['/etc/hive/3.1.0.0-78/0/hive-env.sh.template'] {'owner': 'hive', 'group': 'hadoop', 'mode': 0755}
2020-05-04 21:18:36,459 - File['/etc/hive/3.1.0.0-78/0/llap-daemon-log4j2.properties'] {'content': InlineTemplate(...), 'owner': 'hive', 'group': 'hadoop', 'mode': 0644}
2020-05-04 21:18:36,461 - File['/etc/hive/3.1.0.0-78/0/llap-cli-log4j2.properties'] {'content': InlineTemplate(...), 'owner': 'hive', 'group': 'hadoop', 'mode': 0644}
2020-05-04 21:18:36,463 - File['/etc/hive/3.1.0.0-78/0/hive-log4j2.properties'] {'content': InlineTemplate(...), 'owner': 'hive', 'group': 'hadoop', 'mode': 0644}
2020-05-04 21:18:36,465 - File['/etc/hive/3.1.0.0-78/0/hive-exec-log4j2.properties'] {'content': InlineTemplate(...), 'owner': 'hive', 'group': 'hadoop', 'mode': 0644}
2020-05-04 21:18:36,466 - File['/etc/hive/3.1.0.0-78/0/beeline-log4j2.properties'] {'content': InlineTemplate(...), 'owner': 'hive', 'group': 'hadoop', 'mode': 0644}
2020-05-04 21:18:36,467 - XmlConfig['beeline-site.xml'] {'owner': 'hive', 'group': 'hadoop', 'mode': 0644, 'conf_dir': '/etc/hive/3.1.0.0-78/0', 'configurations': {'beeline.hs2.jdbc.url.container': u'jdbc:hive2://(SERVER).domain:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2', 'beeline.hs2.jdbc.url.default': u'container'}}
2020-05-04 21:18:36,474 - Generating config: /etc/hive/3.1.0.0-78/0/beeline-site.xml
2020-05-04 21:18:36,474 - File['/etc/hive/3.1.0.0-78/0/beeline-site.xml'] {'owner': 'hive', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2020-05-04 21:18:36,476 - File['/etc/hive/3.1.0.0-78/0/parquet-logging.properties'] {'content': ..., 'owner': 'hive', 'group': 'hadoop', 'mode': 0644}
2020-05-04 21:18:36,476 - Directory['/etc/hive/3.1.0.0-78/0'] {'owner': 'hive', 'group': 'hadoop', 'create_parents': True, 'mode': 0755}
2020-05-04 21:18:36,476 - XmlConfig['mapred-site.xml'] {'group': 'hadoop', 'conf_dir': '/etc/hive/3.1.0.0-78/0', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'hive', 'configurations': ...}
2020-05-04 21:18:36,483 - Generating config: /etc/hive/3.1.0.0-78/0/mapred-site.xml
2020-05-04 21:18:36,483 - File['/etc/hive/3.1.0.0-78/0/mapred-site.xml'] {'owner': 'hive', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2020-05-04 21:18:36,516 - File['/etc/hive/3.1.0.0-78/0/hive-default.xml.template'] {'owner': 'hive', 'group': 'hadoop', 'mode': 0644}
2020-05-04 21:18:36,516 - File['/etc/hive/3.1.0.0-78/0/hive-env.sh.template'] {'owner': 'hive', 'group': 'hadoop', 'mode': 0755}
2020-05-04 21:18:36,519 - File['/etc/hive/3.1.0.0-78/0/llap-daemon-log4j2.properties'] {'content': InlineTemplate(...), 'owner': 'hive', 'group': 'hadoop', 'mode': 0644}
2020-05-04 21:18:36,521 - File['/etc/hive/3.1.0.0-78/0/llap-cli-log4j2.properties'] {'content': InlineTemplate(...), 'owner': 'hive', 'group': 'hadoop', 'mode': 0644}
2020-05-04 21:18:36,523 - File['/etc/hive/3.1.0.0-78/0/hive-log4j2.properties'] {'content': InlineTemplate(...), 'owner': 'hive', 'group': 'hadoop', 'mode': 0644}
2020-05-04 21:18:36,525 - File['/etc/hive/3.1.0.0-78/0/hive-exec-log4j2.properties'] {'content': InlineTemplate(...), 'owner': 'hive', 'group': 'hadoop', 'mode': 0644}
2020-05-04 21:18:36,527 - File['/etc/hive/3.1.0.0-78/0/beeline-log4j2.properties'] {'content': InlineTemplate(...), 'owner': 'hive', 'group': 'hadoop', 'mode': 0644}
2020-05-04 21:18:36,527 - XmlConfig['beeline-site.xml'] {'owner': 'hive', 'group': 'hadoop', 'mode': 0644, 'conf_dir': '/etc/hive/3.1.0.0-78/0', 'configurations': {'beeline.hs2.jdbc.url.container': u'jdbc:hive2://(SERVER).domain:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2', 'beeline.hs2.jdbc.url.default': u'container'}}
2020-05-04 21:18:36,534 - Generating config: /etc/hive/3.1.0.0-78/0/beeline-site.xml
2020-05-04 21:18:36,534 - File['/etc/hive/3.1.0.0-78/0/beeline-site.xml'] {'owner': 'hive', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2020-05-04 21:18:36,536 - File['/etc/hive/3.1.0.0-78/0/parquet-logging.properties'] {'content': ..., 'owner': 'hive', 'group': 'hadoop', 'mode': 0644}
2020-05-04 21:18:36,536 - File['/usr/hdp/current/hive-server2/conf/hive-site.jceks'] {'content': StaticFile('/var/lib/ambari-agent/cred/conf/hive_server/hive-site.jceks'), 'owner': 'hive', 'group': 'hadoop', 'mode': 0640}
2020-05-04 21:18:36,537 - Writing File['/usr/hdp/current/hive-server2/conf/hive-site.jceks'] because contents don't match
2020-05-04 21:18:36,537 - XmlConfig['hive-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hive-server2/conf/', 'mode': 0644, 'configuration_attributes': {u'hidden': {u'javax.jdo.option.ConnectionPassword': u'HIVE_CLIENT,CONFIG_DOWNLOAD'}}, 'owner': 'hive', 'configurations': ...}
2020-05-04 21:18:36,543 - Generating config: /usr/hdp/current/hive-server2/conf/hive-site.xml
2020-05-04 21:18:36,544 - File['/usr/hdp/current/hive-server2/conf/hive-site.xml'] {'owner': 'hive', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2020-05-04 21:18:36,675 - File['/usr/hdp/current/hive-server2/conf//hive-env.sh'] {'content': InlineTemplate(...), 'owner': 'hive', 'group': 'hadoop', 'mode': 0755}
2020-05-04 21:18:36,675 - Directory['/etc/security/limits.d'] {'owner': 'root', 'create_parents': True, 'group': 'root'}
2020-05-04 21:18:36,678 - File['/etc/security/limits.d/hive.conf'] {'content': Template('hive.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644}
2020-05-04 21:18:36,678 - File['/usr/lib/ambari-agent/DBConnectionVerification.jar'] {'content': DownloadSource('http://(SERVER):8080/resources/DBConnectionVerification.jar'), 'mode': 0644}
2020-05-04 21:18:36,678 - Not downloading the file from http://(SERVER):8080/resources/DBConnectionVerification.jar, because /var/lib/ambari-agent/tmp/DBConnectionVerification.jar already exists
2020-05-04 21:18:36,679 - Directory['/var/run/hive'] {'owner': 'hive', 'create_parents': True, 'group': 'hadoop', 'mode': 0755, 'cd_access': 'a'}
2020-05-04 21:18:36,679 - Directory['/var/log/hive'] {'owner': 'hive', 'create_parents': True, 'group': 'hadoop', 'mode': 0755, 'cd_access': 'a'}
2020-05-04 21:18:36,679 - Directory['/var/lib/hive'] {'owner': 'hive', 'create_parents': True, 'group': 'hadoop', 'mode': 0755, 'cd_access': 'a'}
2020-05-04 21:18:36,681 - File['/var/lib/ambari-agent/tmp/start_hiveserver2_script'] {'content': Template('startHiveserver2.sh.j2'), 'mode': 0755}
2020-05-04 21:18:36,685 - File['/usr/hdp/current/hive-server2/conf/hadoop-metrics2-hiveserver2.properties'] {'content': Template('hadoop-metrics2-hiveserver2.properties.j2'), 'owner': 'hive', 'group': 'hadoop', 'mode': 0600}
2020-05-04 21:18:36,685 - XmlConfig['hiveserver2-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hive-server2/conf/', 'mode': 0600, 'configuration_attributes': {}, 'owner': 'hive', 'configurations': ...}
2020-05-04 21:18:36,691 - Generating config: /usr/hdp/current/hive-server2/conf/hiveserver2-site.xml
2020-05-04 21:18:36,692 - File['/usr/hdp/current/hive-server2/conf/hiveserver2-site.xml'] {'owner': 'hive', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0600, 'encoding': 'UTF-8'}
2020-05-04 21:18:36,699 - Called copy_to_hdfs tarball: mapreduce
2020-05-04 21:18:36,700 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.0.0-78 -> 3.1.0.0-78
2020-05-04 21:18:36,700 - Tarball version was calcuated as 3.1.0.0-78. Use Command Version: True
2020-05-04 21:18:36,700 - Source file: /usr/hdp/3.1.0.0-78/hadoop/mapreduce.tar.gz , Dest file in HDFS: /hdp/apps/3.1.0.0-78/mapreduce/mapreduce.tar.gz
2020-05-04 21:18:36,700 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.0.0-78 -> 3.1.0.0-78
2020-05-04 21:18:36,700 - Tarball version was calcuated as 3.1.0.0-78. Use Command Version: True
2020-05-04 21:18:36,700 - HdfsResource['/hdp/apps/3.1.0.0-78/mapreduce'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/3.1.0.0-78/hadoop/bin', 'keytab': [EMPTY], 'dfs_type': 'HDFS', 'default_fs': 'hdfs://(SERVER).domain:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': 'missing_principal', 'user': 'hdfs', 'owner': 'hdfs', 'hadoop_conf_dir': '/usr/hdp/3.1.0.0-78/hadoop/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/mr-history/done', u'/warehouse/tablespace/managed/hive', u'/warehouse/tablespace/external/hive', u'/app-logs', u'/tmp'], 'mode': 0555}
2020-05-04 21:18:36,704 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET -d '"'"''"'"' -H '"'"'Content-Length: 0'"'"' '"'"'http://(SERVER).domain:50070/webhdfs/v1/hdp/apps/3.1.0.0-78/mapreduce?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmprKvGIr 2>/tmp/tmpEY8RQP''] {'logoutput': None, 'quiet': False}
2020-05-04 21:18:36,779 - call returned (0, '')
2020-05-04 21:18:36,780 - get_user_call_output returned (0, u'{"FileStatus":{"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":16402,"group":"hdfs","length":0,"modificationTime":1588612036202,"owner":"hdfs","pathSuffix":"","permission":"555","replication":0,"storagePolicy":0,"type":"DIRECTORY"}}200', u'')
2020-05-04 21:18:36,781 - HdfsResource['/hdp/apps/3.1.0.0-78/mapreduce/mapreduce.tar.gz'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/3.1.0.0-78/hadoop/bin', 'keytab': [EMPTY], 'source': '/usr/hdp/3.1.0.0-78/hadoop/mapreduce.tar.gz', 'dfs_type': 'HDFS', 'default_fs': 'hdfs://(SERVER).domain:8020', 'replace_existing_files': False, 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': 'missing_principal', 'user': 'hdfs', 'owner': 'hdfs', 'group': 'hadoop', 'hadoop_conf_dir': '/usr/hdp/3.1.0.0-78/hadoop/conf', 'type': 'file', 'action': ['create_on_execute'], 'immutable_paths': [u'/mr-history/done', u'/warehouse/tablespace/managed/hive', u'/warehouse/tablespace/external/hive', u'/app-logs', u'/tmp'], 'mode': 0444}
2020-05-04 21:18:36,782 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET -d '"'"''"'"' -H '"'"'Content-Length: 0'"'"' '"'"'http://(SERVER).domain:50070/webhdfs/v1/hdp/apps/3.1.0.0-78/mapreduce/mapreduce.tar.gz?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpznS8WR 2>/tmp/tmprj2kz3''] {'logoutput': None, 'quiet': False}
2020-05-04 21:18:36,838 - call returned (0, '')
2020-05-04 21:18:36,838 - get_user_call_output returned (0, u'{"RemoteException":{"exception":"FileNotFoundException","javaClassName":"java.io.FileNotFoundException","message":"File does not exist: /hdp/apps/3.1.0.0-78/mapreduce/mapreduce.tar.gz"}}404', u'')
2020-05-04 21:18:36,839 - Creating new file /hdp/apps/3.1.0.0-78/mapreduce/mapreduce.tar.gz in DFS
2020-05-04 21:18:36,840 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X PUT --data-binary @/usr/hdp/3.1.0.0-78/hadoop/mapreduce.tar.gz -H '"'"'Content-Type: application/octet-stream'"'"' '"'"'http://(SERVER).domain:50070/webhdfs/v1/hdp/apps/3.1.0.0-78/mapreduce/mapreduce.tar.gz?op=CREATE&user.name=hdfs&overwrite=True&permission=444'"'"' 1>/tmp/tmpHOoLlW 2>/tmp/tmpVGvaUr''] {'logoutput': None, 'quiet': False}
2020-05-04 21:18:39,781 - call returned (0, '')
2020-05-04 21:18:39,782 - get_user_call_output returned (0, u'<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>502 Bad Gateway</title>\n</head><body>\n<h1>Bad Gateway</h1>\n<p>The proxy server received an invalid\r\nresponse from an upstream server.<br />\r\n</p>\n</body></html>\n502', u'')

Command failed after 1 tries

 

Thanks in Advance,

 

Cheers,
Chris

1 REPLY 1
Highlighted

Re: Components won't start

Explorer

More on this, I can see in the ambari-server log, each time i start on of those components, i get this message:

 

2020-05-06 19:06:21,593 ERROR [agent-report-processor-0] HeartbeatProcessor:516 - Operation failed - may be retried. Service component host: HISTORYSERVER, 

2020-05-06 19:07:04,696 ERROR [agent-report-processor-0] HeartbeatProcessor:516 - Operation failed - may be retried. Service component host: HIVE_SERVER

 

and every 20 mins this message: 

2020-05-06 19:07:13,420 ERROR [ambari-metrics-retrieval-service-thread-1063] MetricsRetrievalService:496 - Unable to retrieve metrics from http://:19888/jmx. Subsequent failures will be suppressed from the log for 20 minutes.
java.net.ConnectException: Connection refused (Connection refused)

 

If i look at the services, ambari metrics is started. But i don't see that port 19888 active in netstat. 

 

 

 

Don't have an account?
Coming from Hortonworks? Activate your account here