Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Ambari fails on starting services

Ambari fails on starting services

New Contributor

I've managed to install the HortonWorks Data Platform using Ambari and I've got some services up & running including HDFS. Services that depend on HDFS are failing to be installed though.

Having a look at the logs, It seems that Ambary tries to create an HDFS directory but for any reason isn't able to do so. See below the last lines of the installation output log file:

2018-01-18 12:04:29,648 - HdfsResource['/apps/hbase/data'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/2.6.4.0-91/hadoop/bin', 'keytab': [EMPTY], 'dfs_type': '', 'default_fs': 'hdfs://xxxx:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'owner': 'hbase', 'hadoop_conf_dir': '/usr/hdp/2.6.4.0-91/hadoop/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/apps/hive/warehouse', u'/mr-history/done', u'/app-logs', u'/tmp']}
2018-01-18 12:04:29,650 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://xxxxx:50070/webhdfs/v1/apps/hbase/data?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpBbAxgy 2>/tmp/tmpArp5uZ''] {'logoutput': None, 'quiet': False}
2018-01-18 12:05:28,775 - call returned (0, '')
Command failed after 1 tries

The first lines of the stacktrace are the following:

  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 250, in _run_command
    raise WebHDFSCallException(err_msg, result_dict)
resource_management.libraries.providers.hdfs_resource.WebHDFSCallException: Execution of 'curl -sS -L -w '%{http_code}' -X GET 'http://dw09hdp01.ods:50070/webhdfs/v1/apps/hbase/data?op=GETFILESTATUS&user.name=hdfs'' returned status_code=503. 

Any ideas?

4 REPLIES 4

Re: Ambari fails on starting services

Super Mentor

@Alberto Rodriguez

Please try to run HDFS Service checks from ambari UI to see if the HDFS is functioning fine or not?

Ambari UI --> HDFS --> Service Actions --> Run Service Check.

If needed then please try restarting the whole HDFS Service and then try installing the services again.

.

However based on the HTTP return code 503 it looks like there is something wrong going on inside the HDFS NameNode.

returned status_code=503. 

.

As we see 503 (Internal Server Error) coming from the Hadoop NameNode WebHDFS response hence definitely there must be a detailed stack trace / error logged inside the NameNode log. Please check and share the NameNode logs when you noticed the above error in ambari UI.

Re: Ambari fails on starting services

New Contributor

Thank you for replying @Jay Kumar SenSharma,

The HDFS Run Service Check is returning the same error:

  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 250, in _run_command
    raise WebHDFSCallException(err_msg, result_dict)
resource_management.libraries.providers.hdfs_resource.WebHDFSCallException: Execution of 'curl -sS -L -w '%{http_code}' -X GET 'http://xxxxx:50070/webhdfs/v1/tmp?op=GETFILESTATUS&user.name=hdfs'' returned status_code=503. 

I've restarted all the HDFS components and you're right the NameNode is not starting properly (although the Ambari interface shows that the service is up & running:

2018-01-18 15:28:45,730 - Execute['/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://xxxxx:8020 -safemode get | grep 'Safe mode is OFF''] {'logoutput': True, 'tries': 115, 'user': 'hdfs', 'try_sleep': 10}
2018-01-18 15:28:47,790 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://xxxxx:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
......
Safe mode is OFF
2018-01-18 15:30:35,887 - HdfsResource['/tmp'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/2.6.4.0-91/hadoop/bin', 'keytab': [EMPTY], 'dfs_type': '', 'default_fs': 'hdfs://xxxxx:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': None, 'user': 'hdfs', 'owner': 'hdfs', 'hadoop_conf_dir': '/usr/hdp/2.6.4.0-91/hadoop/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/apps/hive/warehouse', u'/mr-history/done', u'/app-logs', u'/tmp'], 'mode': 0777}
2018-01-18 15:30:35,894 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://xxxxx:50070/webhdfs/v1/tmp?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmp75FTX4 2>/tmp/tmpdq8xGk''] {'logoutput': None, 'quiet': False}
2018-01-18 15:31:35,904 - call returned (0, '')

It might be a problem with the curl command executed from the python scripts behind a proxy??

Re: Ambari fails on starting services

New Contributor

Hi all,

I've found out the problem, the calls to the HDFS API went through a proxy and after that they weren't able to resolve my HDFS master host. To fix this I asked to the Ops guys to disabled the proxy and in the meantime I added the --noproxy option to the python script that execute the curl command (/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py) modifying the following line of the script:

cmd = ["curl", "--noproxy", "*", "-sS","-L", "-w", "%{http_code}", "-X", method]

I hope this answer will help someone who find the same problem in the future ;)

I'll create a jira issue and a PR to the Ambari code to include this option to the command.

Re: Ambari fails on starting services

Expert Contributor
@Alberto Rodriguez

Even i got the same issue , i just disabled the proxy and started the service. And running curl with "-noproxy" is better option, i will try that.