Support Questions
Find answers, ask questions, and share your expertise

Unable to start resource manager

Hello,

All datanode are running fine, secondary namenode also running, active namenode is running but not starting through ambari UI. And while starting yarn resource manager giving below output,

2018-05-16 06:33:54,704 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2018-05-16 06:33:54,880 - Stack Feature Version Info: stack_version=2.6, version=2.6.3.0-235, current_cluster_version=2.6.3.0-235 -> 2.6.3.0-235
2018-05-16 06:33:54,889 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
User Group mapping (user_group) is missing in the hostLevelParams
2018-05-16 06:33:54,891 - Group['hadoop'] {}
2018-05-16 06:33:54,892 - Group['users'] {}
2018-05-16 06:33:54,892 - User['infra-solr'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-05-16 06:33:54,894 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-05-16 06:33:54,895 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-05-16 06:33:54,896 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2018-05-16 06:33:54,896 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-05-16 06:33:54,897 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-05-16 06:33:54,897 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-05-16 06:33:54,898 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-05-16 06:33:54,899 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2018-05-16 06:33:54,900 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2018-05-16 06:33:54,951 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2018-05-16 06:33:54,952 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'}
2018-05-16 06:33:54,953 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2018-05-16 06:33:54,954 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2018-05-16 06:33:55,003 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if
2018-05-16 06:33:55,004 - Group['hdfs'] {}
2018-05-16 06:33:55,004 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': [u'hadoop', u'hdfs']}
2018-05-16 06:33:55,005 - FS Type: 
2018-05-16 06:33:55,005 - Directory['/etc/hadoop'] {'mode': 0755}
2018-05-16 06:33:55,027 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2018-05-16 06:33:55,028 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2018-05-16 06:33:55,044 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2018-05-16 06:33:55,091 - Skipping Execute[('setenforce', '0')] due to not_if
2018-05-16 06:33:55,091 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
2018-05-16 06:33:55,095 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
2018-05-16 06:33:55,096 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}
2018-05-16 06:33:55,100 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2018-05-16 06:33:55,102 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2018-05-16 06:33:55,110 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2018-05-16 06:33:55,121 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2018-05-16 06:33:55,122 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2018-05-16 06:33:55,123 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2018-05-16 06:33:55,127 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'}
2018-05-16 06:33:55,171 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2018-05-16 06:33:55,483 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2018-05-16 06:33:55,485 - call['ambari-python-wrap /usr/bin/hdp-select status hadoop-yarn-resourcemanager'] {'timeout': 20}
2018-05-16 06:33:55,556 - call returned (0, 'hadoop-yarn-resourcemanager - 2.6.3.0-235')
2018-05-16 06:33:55,564 - Stack Feature Version Info: stack_version=2.6, version=2.6.3.0-235, current_cluster_version=2.6.3.0-235 -> 2.6.3.0-235
2018-05-16 06:33:55,588 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2018-05-16 06:33:55,605 - Directory['/var/log/hadoop-yarn/nodemanager/recovery-state'] {'owner': 'yarn', 'group': 'hadoop', 'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2018-05-16 06:33:55,607 - Directory['/var/run/hadoop-yarn'] {'owner': 'yarn', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2018-05-16 06:33:55,608 - Directory['/var/run/hadoop-yarn/yarn'] {'owner': 'yarn', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2018-05-16 06:33:55,609 - Directory['/var/log/hadoop-yarn/yarn'] {'owner': 'yarn', 'group': 'hadoop', 'create_parents': True, 'cd_access': 'a'}
2018-05-16 06:33:55,609 - Directory['/var/run/hadoop-mapreduce'] {'owner': 'mapred', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2018-05-16 06:33:55,610 - Directory['/var/run/hadoop-mapreduce/mapred'] {'owner': 'mapred', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2018-05-16 06:33:55,610 - Directory['/var/log/hadoop-mapreduce'] {'owner': 'mapred', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2018-05-16 06:33:55,611 - Directory['/var/log/hadoop-mapreduce/mapred'] {'owner': 'mapred', 'group': 'hadoop', 'create_parents': True, 'cd_access': 'a'}
2018-05-16 06:33:55,612 - Directory['/var/log/hadoop-yarn'] {'owner': 'yarn', 'group': 'hadoop', 'ignore_failures': True, 'create_parents': True, 'cd_access': 'a'}
2018-05-16 06:33:55,612 - XmlConfig['core-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {u'final': {u'fs.defaultFS': u'true'}}, 'owner': 'hdfs', 'configurations': ...}
2018-05-16 06:33:55,622 - Generating config: /usr/hdp/current/hadoop-client/conf/core-site.xml
2018-05-16 06:33:55,622 - File['/usr/hdp/current/hadoop-client/conf/core-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2018-05-16 06:33:55,637 - XmlConfig['hdfs-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {u'final': {u'dfs.support.append': u'true', u'dfs.datanode.data.dir': u'true', u'dfs.namenode.http-address': u'true', u'dfs.namenode.name.dir': u'true', u'dfs.webhdfs.enabled': u'true', u'dfs.datanode.failed.volumes.tolerated': u'true'}}, 'owner': 'hdfs', 'configurations': ...}
2018-05-16 06:33:55,646 - Generating config: /usr/hdp/current/hadoop-client/conf/hdfs-site.xml
2018-05-16 06:33:55,646 - File['/usr/hdp/current/hadoop-client/conf/hdfs-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2018-05-16 06:33:55,691 - XmlConfig['mapred-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'yarn', 'configurations': ...}
2018-05-16 06:33:55,699 - Generating config: /usr/hdp/current/hadoop-client/conf/mapred-site.xml
2018-05-16 06:33:55,700 - File['/usr/hdp/current/hadoop-client/conf/mapred-site.xml'] {'owner': 'yarn', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2018-05-16 06:33:55,737 - Changing owner for /usr/hdp/current/hadoop-client/conf/mapred-site.xml from 1007 to yarn
2018-05-16 06:33:55,738 - XmlConfig['yarn-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'yarn', 'configurations': ...}
2018-05-16 06:33:55,746 - Generating config: /usr/hdp/current/hadoop-client/conf/yarn-site.xml
2018-05-16 06:33:55,747 - File['/usr/hdp/current/hadoop-client/conf/yarn-site.xml'] {'owner': 'yarn', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2018-05-16 06:33:55,839 - XmlConfig['capacity-scheduler.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'yarn', 'configurations': ...}
2018-05-16 06:33:55,847 - Generating config: /usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml
2018-05-16 06:33:55,848 - File['/usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml'] {'owner': 'yarn', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2018-05-16 06:33:55,860 - Changing owner for /usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml from 1005 to yarn
2018-05-16 06:33:55,860 - Directory['/etc/hadoop/conf'] {'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2018-05-16 06:33:55,861 - File['/etc/hadoop/conf/yarn.exclude'] {'owner': 'yarn', 'group': 'hadoop'}
2018-05-16 06:33:55,861 - File['/var/log/hadoop-yarn/yarn/hadoop-mapreduce.jobsummary.log'] {'owner': 'yarn', 'group': 'hadoop'}
2018-05-16 06:33:55,864 - File['/etc/security/limits.d/yarn.conf'] {'content': Template('yarn.conf.j2'), 'mode': 0644}
2018-05-16 06:33:55,867 - File['/etc/security/limits.d/mapreduce.conf'] {'content': Template('mapreduce.conf.j2'), 'mode': 0644}
2018-05-16 06:33:55,873 - File['/usr/hdp/current/hadoop-client/conf/yarn-env.sh'] {'content': InlineTemplate(...), 'owner': 'yarn', 'group': 'hadoop', 'mode': 0755}
2018-05-16 06:33:55,874 - File['/usr/hdp/current/hadoop-yarn-resourcemanager/bin/container-executor'] {'group': 'hadoop', 'mode': 02050}
2018-05-16 06:33:55,876 - File['/usr/hdp/current/hadoop-client/conf/container-executor.cfg'] {'content': Template('container-executor.cfg.j2'), 'group': 'hadoop', 'mode': 0644}
2018-05-16 06:33:55,877 - Directory['/cgroups_test/cpu'] {'group': 'hadoop', 'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2018-05-16 06:33:55,878 - File['/usr/hdp/current/hadoop-client/conf/mapred-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'mode': 0755}
2018-05-16 06:33:55,881 - File['/usr/hdp/current/hadoop-client/conf/taskcontroller.cfg'] {'content': Template('taskcontroller.cfg.j2'), 'owner': 'hdfs'}
2018-05-16 06:33:55,882 - XmlConfig['mapred-site.xml'] {'owner': 'mapred', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2018-05-16 06:33:55,891 - Generating config: /usr/hdp/current/hadoop-client/conf/mapred-site.xml
2018-05-16 06:33:55,891 - File['/usr/hdp/current/hadoop-client/conf/mapred-site.xml'] {'owner': 'mapred', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2018-05-16 06:33:55,929 - Changing owner for /usr/hdp/current/hadoop-client/conf/mapred-site.xml from 1006 to mapred
2018-05-16 06:33:55,930 - XmlConfig['capacity-scheduler.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2018-05-16 06:33:55,938 - Generating config: /usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml
2018-05-16 06:33:55,939 - File['/usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2018-05-16 06:33:55,951 - Changing owner for /usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml from 1006 to hdfs
2018-05-16 06:33:55,951 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2018-05-16 06:33:55,960 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-client.xml
2018-05-16 06:33:55,960 - File['/usr/hdp/current/hadoop-client/conf/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2018-05-16 06:33:55,966 - Directory['/usr/hdp/current/hadoop-client/conf/secure'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2018-05-16 06:33:55,967 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf/secure', 'configuration_attributes': {}, 'configurations': ...}
2018-05-16 06:33:55,976 - Generating config: /usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml
2018-05-16 06:33:55,976 - File['/usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2018-05-16 06:33:55,982 - XmlConfig['ssl-server.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2018-05-16 06:33:55,990 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-server.xml
2018-05-16 06:33:55,991 - File['/usr/hdp/current/hadoop-client/conf/ssl-server.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2018-05-16 06:33:55,997 - File['/usr/hdp/current/hadoop-client/conf/ssl-client.xml.example'] {'owner': 'mapred', 'group': 'hadoop'}
2018-05-16 06:33:55,998 - File['/usr/hdp/current/hadoop-client/conf/ssl-server.xml.example'] {'owner': 'mapred', 'group': 'hadoop'}
2018-05-16 06:33:55,998 - Verifying DFS directories where ATS stores time line data for active and completed applications.
2018-05-16 06:33:55,999 - Verifying if DFS directory '/ats/done/' exists.
2018-05-16 06:33:56,000 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://ambarihdp-produp:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpaGybXZ 2>/tmp/tmpxgj6vG''] {'logoutput': None, 'quiet': False}
2018-05-16 06:33:56,062 - call returned (1, 'su: System error')
2018-05-16 06:33:56,063 - Will retry 7 time(s), caught exception: Execution of 'curl -sS -L -w '%{http_code}' -X GET 'http://ambarihdp-produp:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=hdfs' 1>/tmp/tmpaGybXZ 2>/tmp/tmpxgj6vG' returned 1. 
. Sleeping for 20 sec(s)
2018-05-16 06:34:16,083 - Verifying if DFS directory '/ats/done/' exists.
2018-05-16 06:34:16,084 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://ambarihdp-produp:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpgVe96D 2>/tmp/tmp9qnDje''] {'logoutput': None, 'quiet': False}
2018-05-16 06:34:16,147 - call returned (1, 'su: System error')
2018-05-16 06:34:16,148 - Will retry 6 time(s), caught exception: Execution of 'curl -sS -L -w '%{http_code}' -X GET 'http://ambarihdp-produp:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=hdfs' 1>/tmp/tmpgVe96D 2>/tmp/tmp9qnDje' returned 1. 
. Sleeping for 20 sec(s)
2018-05-16 06:34:36,161 - Verifying if DFS directory '/ats/done/' exists.
2018-05-16 06:34:36,162 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://ambarihdp-produp:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpxKuS14 2>/tmp/tmpAIsP5n''] {'logoutput': None, 'quiet': False}
2018-05-16 06:34:36,224 - call returned (1, 'su: System error')
2018-05-16 06:34:36,225 - Will retry 5 time(s), caught exception: Execution of 'curl -sS -L -w '%{http_code}' -X GET 'http://ambarihdp-produp:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=hdfs' 1>/tmp/tmpxKuS14 2>/tmp/tmpAIsP5n' returned 1. 
. Sleeping for 20 sec(s)
2018-05-16 06:34:56,245 - Verifying if DFS directory '/ats/done/' exists.
2018-05-16 06:34:56,247 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://ambarihdp-produp:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpIOEqT6 2>/tmp/tmpq35ydX''] {'logoutput': None, 'quiet': False}
2018-05-16 06:34:56,311 - call returned (1, 'su: System error')
2018-05-16 06:34:56,311 - Will retry 4 time(s), caught exception: Execution of 'curl -sS -L -w '%{http_code}' -X GET 'http://ambarihdp-produp:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=hdfs' 1>/tmp/tmpIOEqT6 2>/tmp/tmpq35ydX' returned 1. 
. Sleeping for 20 sec(s)
2018-05-16 06:35:16,329 - Verifying if DFS directory '/ats/done/' exists.
2018-05-16 06:35:16,331 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://ambarihdp-produp:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpLvL1_Q 2>/tmp/tmpBLDguy''] {'logoutput': None, 'quiet': False}
2018-05-16 06:35:16,393 - call returned (1, 'su: System error')
2018-05-16 06:35:16,394 - Will retry 3 time(s), caught exception: Execution of 'curl -sS -L -w '%{http_code}' -X GET 'http://ambarihdp-produp:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=hdfs' 1>/tmp/tmpLvL1_Q 2>/tmp/tmpBLDguy' returned 1. 
. Sleeping for 20 sec(s)
2018-05-16 06:35:36,414 - Verifying if DFS directory '/ats/done/' exists.
2018-05-16 06:35:36,416 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://ambarihdp-produp:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpuH3b30 2>/tmp/tmpPfGeOG''] {'logoutput': None, 'quiet': False}
2018-05-16 06:35:36,481 - call returned (1, 'su: System error')
2018-05-16 06:35:36,482 - Will retry 2 time(s), caught exception: Execution of 'curl -sS -L -w '%{http_code}' -X GET 'http://ambarihdp-produp:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=hdfs' 1>/tmp/tmpuH3b30 2>/tmp/tmpPfGeOG' returned 1. 
. Sleeping for 20 sec(s)
2018-05-16 06:35:56,502 - Verifying if DFS directory '/ats/done/' exists.
2018-05-16 06:35:56,504 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://ambarihdp-produp:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpV9TW80 2>/tmp/tmp7JA2AZ''] {'logoutput': None, 'quiet': False}
2018-05-16 06:35:56,567 - call returned (1, 'su: System error')
2018-05-16 06:35:56,567 - Will retry 1 time(s), caught exception: Execution of 'curl -sS -L -w '%{http_code}' -X GET 'http://ambarihdp-produp:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=hdfs' 1>/tmp/tmpV9TW80 2>/tmp/tmp7JA2AZ' returned 1. 
. Sleeping for 20 sec(s)
2018-05-16 06:36:16,587 - Verifying if DFS directory '/ats/done/' exists.
2018-05-16 06:36:16,589 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://ambarihdp-produp:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmplG_OMC 2>/tmp/tmpnGjQfe''] {'logoutput': None, 'quiet': False}
2018-05-16 06:36:16,651 - call returned (1, 'su: System error')

Command failed after 1 tries

And after checking:

#su hdfs

#hdfs fsck /

see below output,

hdfs fsck / Connecting to namenode via http://ambarihdp-produp:50070/fsck?ugi=hdfs&path=%2F FSCK started by hdfs (auth:SIMPLE) from /192.168.1.171 for path / at Wed May 16 06:57:47 UTC 2018 .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... ................................................................................................ /usr/local/share/packages/hadoop/yarn/local/usercache/hdfs/appcache/application_1513432526508_0125/container_e04_1513432526508_0125_01_000001/tmp/temporary-c2ec450d-981e-4b43-ae8b-544c53fdd879/offsets/117: CORRUPT blockpool BP-1214372815-192.168.1.171-1513415035171 block blk_1073803096 /usr/local/share/packages/hadoop/yarn/local/usercache/hdfs/appcache/application_1513432526508_0125/container_e04_1513432526508_0125_01_000001/tmp/temporary-c2ec450d-981e-4b43-ae8b-544c53fdd879/offsets/117: MISSING 1 blocks of total size 224 B.. /usr/local/share/packages/hadoop/yarn/local/usercache/hdfs/appcache/application_1513432526508_0125/container_e04_1513432526508_0125_01_000001/tmp/temporary-c2ec450d-981e-4b43-ae8b-544c53fdd879/offsets/118: CORRUPT blockpool BP-1214372815-192.168.1.171-1513415035171 block blk_1073803114 /usr/local/share/packages/hadoop/yarn/local/usercache/hdfs/appcache/application_1513432526508_0125/container_e04_1513432526508_0125_01_000001/tmp/temporary-c2ec450d-981e-4b43-ae8b-544c53fdd879/offsets/118: MISSING 1 blocks of total size 224 B.... .................................................................................................... .................................................................................................... ..................................................................................... /usr/local/share/packages/hadoop/yarn/local/usercache/hdfs/appcache/application_1513432526508_0128/container_e04_1513432526508_0128_02_000001/tmp/temporary-62a510a1-d20f-47d8-9e68-1c1b7b7b3fc7/commits/0: CORRUPT blockpool BP-1214372815-192.168.1.171-1513415035171 block blk_1073804149 /usr/local/share/packages/hadoop/yarn/local/usercache/hdfs/appcache/application_1513432526508_0128/container_e04_1513432526508_0128_02_000001/tmp/temporary-62a510a1-d20f-47d8-9e68-1c1b7b7b3fc7/commits/0: MISSING 1 blocks of total size 5 B... /usr/local/share/packages/hadoop/yarn/local/usercache/hdfs/appcache/application_1513432526508_0128/container_e04_1513432526508_0128_02_000001/tmp/temporary-62a510a1-d20f-47d8-9e68-1c1b7b7b3fc7/offsets/0: CORRUPT blockpool BP-1214372815-192.168.1.171-1513415035171 block blk_1073804147 /usr/local/share/packages/hadoop/yarn/local/usercache/hdfs/appcache/application_1513432526508_0128/container_e04_1513432526508_0128_02_000001/tmp/temporary-62a510a1-d20f-47d8-9e68-1c1b7b7b3fc7/offsets/0: MISSING 1 blocks of total size 228 B.............. .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .....................................Status: CORRUPT Total size:70566082461 B Total dirs:9474 Total files:45637 Total symlinks:0 Total blocks (validated):45927 (avg. block size 1536483 B) ******************************** UNDER MIN REPL'D BLOCKS:4 (0.008709474 %) dfs.namenode.replication.min:1 CORRUPT FILES:4 MISSING BLOCKS:4 MISSING SIZE:681 B CORRUPT BLOCKS: 4 ******************************** Minimally replicated blocks:45923 (99.99129 %) Over-replicated blocks:0 (0.0 %) Under-replicated blocks:0 (0.0 %) Mis-replicated blocks:0 (0.0 %) Default replication factor:3 Average block replication:2.9997387 Corrupt blocks:4 Missing replicas:0 (0.0 %) Number of data-nodes:6 Number of racks:1 FSCK ended at Wed May 16 06:57:49 UTC 2018 in 1525 milliseconds The filesystem under path '/' is CORRUPT

4 REPLIES 4

getting below error while starting resource manager,

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py", line 244, in <module>
    Resourcemanager().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py", line 124, in start
    self.wait_for_dfs_directories_created(params.entity_groupfs_store_dir, params.entity_groupfs_active_dir)
  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py", line 201, in wait_for_dfs_directories_created
    self.wait_for_dfs_directory_created(dir_path, ignored_dfs_dirs)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/decorator.py", line 62, in wrapper
    return function(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py", line 223, in wait_for_dfs_directory_created
    list_status = util.run_command(dir_path, 'GETFILESTATUS', method='GET', ignore_status_codes=['404'], assertable_result=False)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 186, in run_command
    _, out, err = get_user_call_output(cmd, user=self.run_user, logoutput=self.logoutput, quiet=False)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/get_user_call_output.py", line 61, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
resource_management.core.exceptions.ExecutionFailed: Execution of 'curl -sS -L -w '%{http_code}' -X GET 'http://ambarihdp-produp:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=hdfs' 1>/tmp/tmpjca4KM 2>/tmp/tmpoEbpj4' returned 1.

@Rajendra Manjunath

Thanks for the response but I have already checked. There is no proxy set.

I have started through terminal but not able to start through ambari UI. And proxy is not set. What else can be the issue?

FYI,

Output of /tmp/tmpjca4KM file is as,

curl: (1) Protocol "'http not supported or disabled in libcurl.

Also I have configured curl with ssl. Still facing same issue.

Expert Contributor
@Ankita Ghate

Disable the proxy on the server and try starting the reasource manager from Ambari.