Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HDFS Service Check Timing Out

HDFS Service Check Timing Out

I'm receiving the following error when running the HDFS service check. It seems to be timing out on a WebHDFS PUT statement...

Ambari Version 2.1.1

Python script has been killed due to timeout after waiting 300 secs
2016-02-08 08:37:18,268 - ExecuteHadoop['dfsadmin -fs hdfs://vxkid-phdpdv05.lmig.com:8020 -safemode get | grep OFF'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'logoutput': True, 'try_sleep': 3, 'tries': 20, 'user': 'hdfs'}
2016-02-08 08:37:18,270 - Execute['hadoop --config /usr/hdp/current/hadoop-client/conf dfsadmin -fs hdfs://vxkid-phdpdv05.lmig.com:8020 -safemode get | grep OFF'] {'logoutput': True, 'try_sleep': 3, 'environment': {}, 'tries': 20, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

Safe mode is OFF
2016-02-08 08:37:21,067 - HdfsResource['/tmp'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'default_fs': 'hdfs://vxkid-phdpdv05.lmig.com:8020', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': None, 'user': 'hdfs', 'action': ['create_on_execute'], 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'mode': 0777}
2016-02-08 08:37:21,071 - checked_call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://vxkid-phdpdv05.lmig.com:50070/webhdfs/v1/tmp?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmp5cvrI_ 2>/tmp/tmpcKO2SB''] {'logoutput': None, 'quiet': False}
2016-02-08 08:37:21,259 - checked_call returned (0, '')
2016-02-08 08:37:21,260 - HdfsResource['/tmp/idb60a9aca_date370816'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'default_fs': 'hdfs://vxkid-phdpdv05.lmig.com:8020', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': None, 'user': 'hdfs', 'action': ['delete_on_execute'], 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'file'}
2016-02-08 08:37:21,261 - checked_call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://vxkid-phdpdv05.lmig.com:50070/webhdfs/v1/tmp/idb60a9aca_date370816?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpggYpl7 2>/tmp/tmpW_lYIG''] {'logoutput': None, 'quiet': False}
2016-02-08 08:37:21,426 - checked_call returned (0, '')
2016-02-08 08:37:21,427 - HdfsResource['/tmp/idb60a9aca_date370816'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'source': '/etc/passwd', 'default_fs': 'hdfs://vxkid-phdpdv05.lmig.com:8020', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': None, 'user': 'hdfs', 'action': ['create_on_execute'], 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'file'}
2016-02-08 08:37:21,428 - checked_call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://vxkid-phdpdv05.lmig.com:50070/webhdfs/v1/tmp/idb60a9aca_date370816?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmp1n4fuh 2>/tmp/tmpwXsc1P''] {'logoutput': None, 'quiet': False}
2016-02-08 08:37:21,593 - checked_call returned (0, '')
2016-02-08 08:37:21,594 - Creating new file /tmp/idb60a9aca_date370816 in DFS
2016-02-08 08:37:21,595 - checked_call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X PUT -T /etc/passwd '"'"'http://vxkid-phdpdv05.lmig.com:50070/webhdfs/v1/tmp/idb60a9aca_date370816?op=CREATE&user.name=hdfs&overwrite=True'"'"' 1>/tmp/tmp9ChlXL 2>/tmp/tmpCmlsMf''] {'logoutput': None, 'quiet': False}
7 REPLIES 7
Highlighted

Re: HDFS Service Check Timing Out

Mentor
@Hugo Schieck

can you run webhdfs commands manually and execute them successfully? If so, you can skip the service checks until later. Replace sandbox FQDN with your namenode URL.

# list the output directory, notice the webhdfs port
curl -i "http://sandbox.hortonworks.com:50070/webhdfs/v1/user/root/output/?op=LISTSTATUS"

# read the output file
curl -i -L "http://sandbox.hortonworks.com:50070/webhdfs/v1/user/root/output/stdout?op=OPEN"

# rename a file, if you get dr. who error, add &user.name=root or any other user in the context
curl -i -X PUT "sandbox.hortonworks.com:50070/webhdfs/v1/user/root/output/stdout?op=RENAME&user.name=root&destination=/user/root/newname"

# read the output of the new file
curl -i -L "http://sandbox.hortonworks.com:50070/webhdfs/v1/user/root/newname?op=OPEN"
Highlighted

Re: HDFS Service Check Timing Out

Mentor
@Hugo Schieck

check the namenode logs, your namenode is probably in safe mode. If it is in safe mode please run the following, as long as you're on sandbox. On real cluster you need safemode to be exited by itself.

hdfs dfsadmin -safeMode leave
Highlighted

Re: HDFS Service Check Timing Out

@Hugo Schieck

Please retry the operation and also , check namenode logs

Highlighted

Re: HDFS Service Check Timing Out

@Hugo Schieck whats agent.package.install.task.timeout in /etc/ambari-server/conf/ambari.properties?

There was a bug in before 2.0.0. https://issues.apache.org/jira/browse/AMBARI-9881

[root@phdns01 ~]# grep agent.package.install.task.timeout /etc/ambari-server/conf/ambari.properties

agent.package.install.task.timeout=1800

[root@phdns01 ~]#

Highlighted

Re: HDFS Service Check Timing Out

Looks to be set properly.

[root@localhost data]# grep agent.package.install.task.timeout /etc/ambari-server/conf/ambari.properties

agent.package.install.task.timeout=1800

[root@localhost data]#

Highlighted

Re: HDFS Service Check Timing Out

Highlighted

Re: HDFS Service Check Timing Out

I'm using Ambari2.1.1 and HDFS Service check is completed successfully.hdfsservice-check.png

Don't have an account?
Coming from Hortonworks? Activate your account here