Support Questions
Find answers, ask questions, and share your expertise

Yarn service check failed

Yarn service check failed

Explorer

Hi, need some help here. I run yarn service check but it failed. I dont have any idea why this thing happened. I use Ambari 2..5.0.3 and HDP 2.6.0.3. Here is the stderr & stdout:

stderr:

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/service_check.py", line 181, in <module>
    ServiceCheck().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 314, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/service_check.py", line 121, in service_check
    app_url = m.group(1)
AttributeError: 'NoneType' object has no attribute 'group'

stdout:

2017-06-07 12:00:54,974 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-06-07 12:00:54,976 - call['ambari-python-wrap /usr/bin/hdp-select status hadoop-yarn-resourcemanager'] {'timeout': 20}
2017-06-07 12:00:55,025 - call returned (0, 'hadoop-yarn-resourcemanager - 2.6.0.3-8')
2017-06-07 12:00:55,035 - Stack Feature Version Info: stack_version=2.6, version=None, current_cluster_version=2.6.0.3-8 -> 2.6
2017-06-07 12:00:55,070 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-06-07 12:00:55,089 - HdfsResource['/user/ambari-qa'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'dfs_type': '', 'default_fs': 'hdfs://HAcluster', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'owner': 'ambari-qa', 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/apps/hive/warehouse', u'/tmp', u'/app-logs', u'/mr-history/done'], 'mode': 0770}
2017-06-07 12:00:55,095 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://namenode1.domain.com:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpVXOAUM 2>/tmp/tmp7cGGXd''] {'quiet': False}
2017-06-07 12:00:55,147 - call returned (0, '')
2017-06-07 12:00:55,148 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://namenode2.domain.com:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp0PkOtV 2>/tmp/tmphGHhrb''] {'quiet': False}
2017-06-07 12:00:55,199 - call returned (0, '')
2017-06-07 12:00:55,201 - NameNode HA states: active_namenodes = [(u'nn1', 'namenode1.domain.com:50070')], standby_namenodes = [(u'nn2', 'namenode2.domain.com:50070')], unknown_namenodes = []
2017-06-07 12:00:55,202 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://namenode1.domain.com:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpFL6kBX 2>/tmp/tmp2fO1Sp''] {'quiet': False}
2017-06-07 12:00:55,253 - call returned (0, '')
2017-06-07 12:00:55,255 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://namenode2.domain.com:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpPsfDz2 2>/tmp/tmpXnEVlq''] {'quiet': False}
2017-06-07 12:00:55,306 - call returned (0, '')
2017-06-07 12:00:55,307 - NameNode HA states: active_namenodes = [(u'nn1', 'namenode1.domain.com:50070')], standby_namenodes = [(u'nn2', 'namenode2.domain.com:50070')], unknown_namenodes = []
2017-06-07 12:00:55,309 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://namenode1.domain.com:50070/webhdfs/v1/user/ambari-qa?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmp4LNXwH 2>/tmp/tmpyleJQc''] {'logoutput': None, 'quiet': False}
2017-06-07 12:00:55,364 - call returned (0, '')
2017-06-07 12:00:55,372 - checked_call['yarn org.apache.hadoop.yarn.applications.distributedshell.Client -shell_command ls -num_containers 1 -jar /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar -timeout 300000 --queue default'] {'path': '/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin', 'user': 'ambari-qa'}
2017-06-07 12:01:06,017 - checked_call returned (0, '')


Command failed after 1 tries
1 REPLY 1

Re: Yarn service check failed

Rising Star

The service check is looking for a particular string (appTrackingUrl=.*) in the output of the command 'yarn org.apache.hadoop.yarn.applications.distributedshell.Client -shell_command ls -num_containers 1 -jar /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar -timeout 300000 --queue default' and this error means that the string was not present. This probably means that the command failed. I suggest looking into why the command failed. You could see if the app appears in the RM UI. You could try to locate the stdout/stderr for the command, and/or try running the command manually as the ambari-qa user to see what the error is.