Created 06-17-2019 11:38 PM
traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HBASE/package/scripts/hbase_master.py", line 170, in <module> HbaseMaster().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HBASE/package/scripts/hbase_master.py", line 87, in start self.configure(env) # for security File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HBASE/package/scripts/hbase_master.py", line 45, in configure hbase(name='master') File "/usr/lib/ambari-agent/lib/ambari_commons/os_family_impl.py", line 89, in thunk return fn(*args, **kwargs) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HBASE/package/scripts/hbase.py", line 224, in hbase owner=params.hbase_user File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 677, in action_create_on_execute self.action_delayed("create") File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 674, in action_delayed self.get_hdfs_resource_executor().action_delayed(action_name, self) File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 373, in action_delayed self.action_delayed_for_nameservice(None, action_name, main_resource) File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 395, in action_delayed_for_nameservice self._assert_valid() File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 334, in _assert_valid self.target_status = self._get_file_status(target) File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 497, in _get_file_status list_status = self.util.run_command(target, 'GETFILESTATUS', method='GET', ignore_status_codes=['404'], assertable_result=False) File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 214, in run_command return self._run_command(*args, **kwargs) File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 282, in _run_command _, out, err = get_user_call_output(cmd, user=self.run_user, logoutput=self.logoutput, quiet=False) File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output raise ExecutionFailed(err_msg, code, files_output[0], files_output[1]) resource_management.core.exceptions.ExecutionFailed: Execution of 'curl -sS -L -w '%{http_code}' -X GET -d '' -H 'Content-Length: 0' 'http://:50070/webhdfs/v1/apps/hbase/data?op=GETFILESTATUS&user.name=hdfs' 1>/tmp/tmpjXBSTF 2>/tmp/tmpzy89Hf' returned 7. curl: (7) Failed connect to :50070; Connection refused 000
Created 06-17-2019 11:46 PM
When you attempt to start the HBase from Ambari UI / API call then Ambari will also check the accessibility and Availability of HDFS
Like Ambari will check this:
1. If the NameNode is UP and running fine or not)
2. If from the HMaster node you are able to make the following kind of NameNode webhdfs call or not?
# curl -X GET "http://<fqdn>:50070/webhdfs/v1/apps/hbase/data?op=GETFILESTATUS&user.name=hdfs"
In your case it seems that NameNode has some issue in responding (may be NameNode is not running or It might not be healthy) hence the following JMX call to namenode fails to respond.
ERROR:
resource_management.core.exceptions.ExecutionFailed: Execution of 'curl -sS -L -w '%{http_code}' -X GET -d '' -H 'Content-Length: 0' 'http://<fqdn>:50070/webhdfs/v1/apps/hbase/data?op=GETFILESTATUS&user.name=hdfs' 1>/tmp/tmpjXBSTF 2>/tmp/tmpzy89Hf' returned 7. curl: (7) Failed connect to <fqdn>:50070; Connection refused
So you will need to first check:
1. If your NameNode is Up and Running fine. Please check the NameNode logs for any errors.
# ps -ef | grep NameNode # less /var/log/hadoop/hdfs/hadoop-hdfs-namenode-*.log
2. Check if NameNode port 50070 is opened and listening.
# netstat -tnlpa | grep 50070
3. Check if the Firewall is disabled on NameNode host so that other hosts can access the port 50070 remotely.
4. From the Other nodes like HBase HMaster node try to telnet or netcat to port 50070 of NameNode host to verify if there are any N/W or firewall issue.
# telnet <namenode_fqdn> 50070 (OR) # nc -v <namenode_fqdn> 50070
.
Created 06-18-2019 03:24 AM
The above was originally posted in the Community Help Track. On Tue Jun 18 03:10 UTC 2019, a member of the HCC moderation staff moved it to the Cloud & Operations track. The Community Help Track is intended for questions about using the HCC site itself.