Support Questions
Find answers, ask questions, and share your expertise

unable to start apptime server in yarn -Ambari Hadoop

unable to start apptime server in yarn -Ambari Hadoop

i am getting error is like that please help me to start the application.

stderr: /var/lib/ambari-agent/data/errors-5587.txt

Python script has been killed due to timeout after waiting 1200 secs

2018-03-30 10:16:47,796 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2018-03-30 10:16:47,916 - Stack Feature Version Info: stack_version=2.6, version=2.6.0.3-8, current_cluster_version=2.6.0.3-8 -> 2.6.0.3-8
2018-03-30 10:16:47,920 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
User Group mapping (user_group) is missing in the hostLevelParams
2018-03-30 10:16:47,921 - Group['livy'] {}
2018-03-30 10:16:47,922 - Group['spark'] {}
2018-03-30 10:16:47,922 - Group['hadoop'] {}
2018-03-30 10:16:47,922 - Group['users'] {}
2018-03-30 10:16:47,922 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-03-30 10:16:47,922 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-03-30 10:16:47,923 - User['infra-solr'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-03-30 10:16:47,923 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2018-03-30 10:16:47,924 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-03-30 10:16:47,924 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2018-03-30 10:16:47,924 - User['livy'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-03-30 10:16:47,925 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-03-30 10:16:47,925 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2018-03-30 10:16:47,925 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-03-30 10:16:47,926 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-03-30 10:16:47,926 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-03-30 10:16:47,926 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-03-30 10:16:47,927 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-03-30 10:16:47,927 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2018-03-30 10:16:47,928 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2018-03-30 10:16:47,931 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2018-03-30 10:16:47,932 - Group['hdfs'] {}
2018-03-30 10:16:47,932 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': [u'hadoop', u'hdfs']}
2018-03-30 10:16:47,932 - FS Type: 
2018-03-30 10:16:47,932 - Directory['/etc/hadoop'] {'mode': 0755}
2018-03-30 10:16:47,942 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2018-03-30 10:16:47,943 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2018-03-30 10:16:47,957 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2018-03-30 10:16:47,960 - Skipping Execute[('setenforce', '0')] due to not_if
2018-03-30 10:16:47,961 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
2018-03-30 10:16:47,962 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
2018-03-30 10:16:47,962 - Changing owner for /var/run/hadoop from 1010 to root
2018-03-30 10:16:47,962 - Changing group for /var/run/hadoop from 1003 to root
2018-03-30 10:16:47,962 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}
2018-03-30 10:16:47,964 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2018-03-30 10:16:47,966 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2018-03-30 10:16:47,970 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2018-03-30 10:16:47,976 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2018-03-30 10:16:47,977 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2018-03-30 10:16:47,977 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2018-03-30 10:16:47,980 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'}
2018-03-30 10:16:47,983 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2018-03-30 10:16:48,167 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2018-03-30 10:16:48,167 - call['ambari-python-wrap /usr/bin/hdp-select status hadoop-yarn-resourcemanager'] {'timeout': 20}
2018-03-30 10:16:48,181 - call returned (0, 'hadoop-yarn-resourcemanager - 2.6.0.3-8')
2018-03-30 10:16:48,185 - Stack Feature Version Info: stack_version=2.6, version=2.6.0.3-8, current_cluster_version=2.6.0.3-8 -> 2.6.0.3-8
2018-03-30 10:16:48,196 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2018-03-30 10:16:48,206 - Directory['/var/log/hadoop-yarn/nodemanager/recovery-state'] {'owner': 'yarn', 'group': 'hadoop', 'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2018-03-30 10:16:48,206 - Directory['/var/run/hadoop-yarn'] {'owner': 'yarn', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2018-03-30 10:16:48,207 - Directory['/var/run/hadoop-yarn/yarn'] {'owner': 'yarn', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2018-03-30 10:16:48,207 - Directory['/var/log/hadoop-yarn/yarn'] {'owner': 'yarn', 'group': 'hadoop', 'create_parents': True, 'cd_access': 'a'}
2018-03-30 10:16:48,207 - Directory['/var/run/hadoop-mapreduce'] {'owner': 'mapred', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2018-03-30 10:16:48,207 - Directory['/var/run/hadoop-mapreduce/mapred'] {'owner': 'mapred', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2018-03-30 10:16:48,208 - Directory['/var/log/hadoop-mapreduce'] {'owner': 'mapred', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2018-03-30 10:16:48,208 - Directory['/var/log/hadoop-mapreduce/mapred'] {'owner': 'mapred', 'group': 'hadoop', 'create_parents': True, 'cd_access': 'a'}
2018-03-30 10:16:48,208 - Directory['/var/log/hadoop-yarn'] {'owner': 'yarn', 'group': 'hadoop', 'ignore_failures': True, 'create_parents': True, 'cd_access': 'a'}
2018-03-30 10:16:48,208 - XmlConfig['core-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {u'final': {u'fs.defaultFS': u'true'}}, 'owner': 'hdfs', 'configurations': ...}
2018-03-30 10:16:48,214 - Generating config: /usr/hdp/current/hadoop-client/conf/core-site.xml
2018-03-30 10:16:48,214 - File['/usr/hdp/current/hadoop-client/conf/core-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2018-03-30 10:16:48,227 - XmlConfig['hdfs-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {u'final': {u'dfs.support.append': u'true', u'dfs.datanode.data.dir': u'true', u'dfs.namenode.http-address': u'true', u'dfs.namenode.name.dir': u'true', u'dfs.webhdfs.enabled': u'true', u'dfs.datanode.failed.volumes.tolerated': u'true'}}, 'owner': 'hdfs', 'configurations': ...}
2018-03-30 10:16:48,232 - Generating config: /usr/hdp/current/hadoop-client/conf/hdfs-site.xml
2018-03-30 10:16:48,232 - File['/usr/hdp/current/hadoop-client/conf/hdfs-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2018-03-30 10:16:48,257 - XmlConfig['mapred-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'yarn', 'configurations': ...}
2018-03-30 10:16:48,262 - Generating config: /usr/hdp/current/hadoop-client/conf/mapred-site.xml
2018-03-30 10:16:48,262 - File['/usr/hdp/current/hadoop-client/conf/mapred-site.xml'] {'owner': 'yarn', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2018-03-30 10:16:48,287 - Changing owner for /usr/hdp/current/hadoop-client/conf/mapred-site.xml from 1012 to yarn
2018-03-30 10:16:48,287 - XmlConfig['yarn-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'yarn', 'configurations': ...}
2018-03-30 10:16:48,292 - Generating config: /usr/hdp/current/hadoop-client/conf/yarn-site.xml
2018-03-30 10:16:48,292 - File['/usr/hdp/current/hadoop-client/conf/yarn-site.xml'] {'owner': 'yarn', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2018-03-30 10:16:48,346 - XmlConfig['capacity-scheduler.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'yarn', 'configurations': ...}
2018-03-30 10:16:48,351 - Generating config: /usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml
2018-03-30 10:16:48,351 - File['/usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml'] {'owner': 'yarn', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2018-03-30 10:16:48,359 - Changing owner for /usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml from 1010 to yarn
2018-03-30 10:16:48,359 - Directory['/hadoop/yarn/timeline'] {'owner': 'yarn', 'group': 'hadoop', 'create_parents': True, 'cd_access': 'a'}
2018-03-30 10:16:48,359 - Directory['/hadoop/yarn/timeline'] {'owner': 'yarn', 'group': 'hadoop', 'create_parents': True, 'cd_access': 'a'}
2018-03-30 10:16:48,359 - HdfsResource['/ats/done'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'dfs_type': '', 'default_fs': 'hdfs://hdpmaster.blpclean.com:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'change_permissions_for_parents': True, 'owner': 'yarn', 'group': 'hadoop', 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/apps/hive/warehouse', u'/mr-history/done', u'/app-logs', u'/tmp'], 'mode': 0755}
2018-03-30 10:16:48,362 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://hdpmaster.blpclean.com:50070/webhdfs/v1/ats/done?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpj2wjUP 2>/tmp/tmpEkIybQ''] {'logoutput': None, 'quiet': False}
2018-03-30 10:16:48,417 - call returned (0, '')
2018-03-30 10:16:48,422 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X PUT '"'"'http://hdpmaster.blpclean.com:50070/webhdfs/v1/ats/done?op=SETPERMISSION&user.name=hdfs&permission=755'"'"' 1>/tmp/tmpR_HIIl 2>/tmp/tmpKyWmV1''] {'logoutput': None, 'quiet': False}
2018-03-30 10:16:48,515 - call returned (0, '')
2018-03-30 10:16:48,538 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X PUT '"'"'http://hdpmaster.blpclean.com:50070/webhdfs/v1/ats/?op=SETPERMISSION&user.name=hdfs&permission=755'"'"' 1>/tmp/tmpfyDYRJ 2>/tmp/tmp885jQz''] {'logoutput': None, 'quiet': False}
2018-03-30 10:16:48,607 - call returned (0, '')
2018-03-30 10:16:48,608 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X PUT '"'"'http://hdpmaster.blpclean.com:50070/webhdfs/v1/ats/done/?op=SETPERMISSION&user.name=hdfs&permission=755'"'"' 1>/tmp/tmpxLB6ZF 2>/tmp/tmpWZNrg8''] {'logoutput': None, 'quiet': False}
2018-03-30 10:16:48,682 - call returned (0, '')
2018-03-30 10:16:48,683 - HdfsResource['/ats/done/'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'dfs_type': '', 'default_fs': 'hdfs://hdpmaster.blpclean.com:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'owner': 'yarn', 'group': 'hadoop', 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/apps/hive/warehouse', u'/mr-history/done', u'/app-logs', u'/tmp'], 'mode': 0700}
2018-03-30 10:16:48,684 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://hdpmaster.blpclean.com:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpUltLvF 2>/tmp/tmp2gHIvy''] {'logoutput': None, 'quiet': False}
2018-03-30 10:16:48,734 - call returned (0, '')
2018-03-30 10:16:48,735 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X PUT '"'"'http://hdpmaster.blpclean.com:50070/webhdfs/v1/ats/done/?op=SETPERMISSION&user.name=hdfs&permission=700'"'"' 1>/tmp/tmpZu0cn7 2>/tmp/tmpKluuMM''] {'logoutput': None, 'quiet': False}
2018-03-30 10:16:48,816 - call returned (0, '')
2018-03-30 10:16:48,816 - HdfsResource['/ats/active'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'dfs_type': '', 'default_fs': 'hdfs://hdpmaster.blpclean.com:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'change_permissions_for_parents': True, 'owner': 'yarn', 'group': 'hadoop', 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/apps/hive/warehouse', u'/mr-history/done', u'/app-logs', u'/tmp'], 'mode': 0755}
2018-03-30 10:16:48,818 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://hdpmaster.blpclean.com:50070/webhdfs/v1/ats/active?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmp_dbEXX 2>/tmp/tmpxb2lCN''] {'logoutput': None, 'quiet': False}
2018-03-30 10:16:48,862 - call returned (0, '')
2018-03-30 10:16:48,864 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X PUT '"'"'http://hdpmaster.blpclean.com:50070/webhdfs/v1/ats/active?op=SETPERMISSION&user.name=hdfs&permission=755'"'"' 1>/tmp/tmp9EWd0E 2>/tmp/tmpCZck2q''] {'logoutput': None, 'quiet': False}
2018-03-30 10:16:48,940 - call returned (0, '')
2018-03-30 10:16:48,942 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X PUT '"'"'http://hdpmaster.blpclean.com:50070/webhdfs/v1/ats/?op=SETPERMISSION&user.name=hdfs&permission=755'"'"' 1>/tmp/tmpfdJwNa 2>/tmp/tmpTF9qIQ''] {'logoutput': None, 'quiet': False}
2018-03-30 10:16:48,998 - call returned (0, '')
2018-03-30 10:16:48,999 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X PUT '"'"'http://hdpmaster.blpclean.com:50070/webhdfs/v1/ats/active/?op=SETPERMISSION&user.name=hdfs&permission=755'"'"' 1>/tmp/tmptiXP3R 2>/tmp/tmp_HpLov''] {'logoutput': None, 'quiet': False}
2018-03-30 10:16:49,057 - call returned (0, '')
2018-03-30 10:16:49,058 - HdfsResource['/ats/active/'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'dfs_type': '', 'default_fs': 'hdfs://hdpmaster.blpclean.com:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'owner': 'yarn', 'group': 'hadoop', 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/apps/hive/warehouse', u'/mr-history/done', u'/app-logs', u'/tmp'], 'mode': 01777}
2018-03-30 10:16:49,059 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://hdpmaster.blpclean.com:50070/webhdfs/v1/ats/active/?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmp_rmmec 2>/tmp/tmptGuVLr''] {'logoutput': None, 'quiet': False}
2018-03-30 10:16:49,096 - call returned (0, '')
2018-03-30 10:16:49,097 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X PUT '"'"'http://hdpmaster.blpclean.com:50070/webhdfs/v1/ats/active/?op=SETPERMISSION&user.name=hdfs&permission=1777'"'"' 1>/tmp/tmpT52N4z 2>/tmp/tmpzP2Du5''] {'logoutput': None, 'quiet': False}
2018-03-30 10:16:49,140 - call returned (0, '')
2018-03-30 10:16:49,141 - HdfsResource[None] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'dfs_type': '', 'default_fs': 'hdfs://hdpmaster.blpclean.com:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'action': ['execute'], 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'immutable_paths': [u'/apps/hive/warehouse', u'/mr-history/done', u'/app-logs', u'/tmp']}
2018-03-30 10:16:49,145 - File['/etc/security/limits.d/yarn.conf'] {'content': Template('yarn.conf.j2'), 'mode': 0644}
2018-03-30 10:16:49,147 - File['/etc/security/limits.d/mapreduce.conf'] {'content': Template('mapreduce.conf.j2'), 'mode': 0644}
2018-03-30 10:16:49,153 - File['/usr/hdp/current/hadoop-client/conf/yarn-env.sh'] {'content': InlineTemplate(...), 'owner': 'yarn', 'group': 'hadoop', 'mode': 0755}
2018-03-30 10:16:49,154 - Writing File['/usr/hdp/current/hadoop-client/conf/yarn-env.sh'] because contents don't match
2018-03-30 10:16:49,155 - File['/usr/hdp/current/hadoop-yarn-timelineserver/bin/container-executor'] {'group': 'hadoop', 'mode': 02050}
2018-03-30 10:16:49,156 - File['/usr/hdp/current/hadoop-client/conf/container-executor.cfg'] {'content': Template('container-executor.cfg.j2'), 'group': 'hadoop', 'mode': 0644}
2018-03-30 10:16:49,157 - Directory['/cgroups_test/cpu'] {'group': 'hadoop', 'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2018-03-30 10:16:49,158 - File['/usr/hdp/current/hadoop-client/conf/mapred-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'mode': 0755}
2018-03-30 10:16:49,159 - File['/usr/hdp/current/hadoop-client/conf/taskcontroller.cfg'] {'content': Template('taskcontroller.cfg.j2'), 'owner': 'hdfs'}
2018-03-30 10:16:49,159 - XmlConfig['mapred-site.xml'] {'owner': 'mapred', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2018-03-30 10:16:49,165 - Generating config: /usr/hdp/current/hadoop-client/conf/mapred-site.xml
2018-03-30 10:16:49,165 - File['/usr/hdp/current/hadoop-client/conf/mapred-site.xml'] {'owner': 'mapred', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2018-03-30 10:16:49,188 - Changing owner for /usr/hdp/current/hadoop-client/conf/mapred-site.xml from 1011 to mapred
2018-03-30 10:16:49,189 - XmlConfig['capacity-scheduler.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2018-03-30 10:16:49,193 - Generating config: /usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml
2018-03-30 10:16:49,193 - File['/usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2018-03-30 10:16:49,200 - Changing owner for /usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml from 1011 to hdfs
2018-03-30 10:16:49,201 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2018-03-30 10:16:49,205 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-client.xml
2018-03-30 10:16:49,206 - File['/usr/hdp/current/hadoop-client/conf/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2018-03-30 10:16:49,209 - Directory['/usr/hdp/current/hadoop-client/conf/secure'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2018-03-30 10:16:49,209 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf/secure', 'configuration_attributes': {}, 'configurations': ...}
2018-03-30 10:16:49,214 - Generating config: /usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml
2018-03-30 10:16:49,214 - File['/usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2018-03-30 10:16:49,217 - XmlConfig['ssl-server.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2018-03-30 10:16:49,222 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-server.xml
2018-03-30 10:16:49,222 - File['/usr/hdp/current/hadoop-client/conf/ssl-server.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2018-03-30 10:16:49,226 - File['/usr/hdp/current/hadoop-client/conf/ssl-client.xml.example'] {'owner': 'mapred', 'group': 'hadoop'}
2018-03-30 10:16:49,226 - File['/usr/hdp/current/hadoop-client/conf/ssl-server.xml.example'] {'owner': 'mapred', 'group': 'hadoop'}
2018-03-30 10:16:49,226 - File['/var/run/hadoop-yarn/yarn/yarn-yarn-timelineserver.pid'] {'action': ['delete'], 'not_if': 'ambari-sudo.sh  -H -E test -f /var/run/hadoop-yarn/yarn/yarn-yarn-timelineserver.pid && ambari-sudo.sh  -H -E pgrep -F /var/run/hadoop-yarn/yarn/yarn-yarn-timelineserver.pid'}
2018-03-30 10:16:49,230 - File['/hadoop/yarn/timeline/leveldb-timeline-store.ldb/LOCK'] {'action': ['delete'], 'not_if': 'ambari-sudo.sh  -H -E test -f /var/run/hadoop-yarn/yarn/yarn-yarn-timelineserver.pid && ambari-sudo.sh  -H -E pgrep -F /var/run/hadoop-yarn/yarn/yarn-yarn-timelineserver.pid', 'ignore_failures': True, 'only_if': 'ls /hadoop/yarn/timeline/leveldb-timeline-store.ldb/LOCK'}
2018-03-30 10:16:49,252 - Skipping File['/hadoop/yarn/timeline/leveldb-timeline-store.ldb/LOCK'] due to only_if
2018-03-30 10:16:49,253 - Execute['ulimit -c unlimited; export HADOOP_LIBEXEC_DIR=/usr/hdp/current/hadoop-client/libexec && /usr/hdp/current/hadoop-yarn-timelineserver/sbin/yarn-daemon.sh --config /usr/hdp/current/hadoop-client/conf start timelineserver'] {'not_if': 'ambari-sudo.sh  -H -E test -f /var/run/hadoop-yarn/yarn/yarn-yarn-timelineserver.pid && ambari-sudo.sh  -H -E pgrep -F /var/run/hadoop-yarn/yarn/yarn-yarn-timelineserver.pid', 'user': 'yarn'}

Command failed after 1 tries

	error i am getting
6 REPLIES 6

Re: unable to start apptime server in yarn -Ambari Hadoop

Expert Contributor

Hi @shrinivas acharya,

Can you check errors in app timeline sever log file?

-Shubham

Re: unable to start apptime server in yarn -Ambari Hadoop

sure, please find the attached apptime line server log.log.txt

Re: unable to start apptime server in yarn -Ambari Hadoop

Expert Contributor

App timeline server is not able to connect to HDFS.

	2018-03-29 15:30:44,338 ERROR timeline.EntityGroupFSTimelineStore (EntityGroupFSTimelineStore.java:run(852)) - Error scanning active files
java.net.ConnectException: Call From hdpmaster.blpclean.com/192.168.1.10 to hdpmaster.blpclean.com:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.GeneratedConstructorAccessor217.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruct
	2018-03-29 15:31:18,323 ERROR applicationhistoryservice.ApplicationHistoryServer (LogAdapter.java:error(69)) - RECEIVED SIGNAL 15: SIGTERM 
	2018-03-29 15:31:28,384 INFO  mortbay.log (Slf4jLog.java:info(67)) - Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:8188

Please check whether HDFS services are up and check whether HDFS port 8020 is accessible from Timeline server.

Re: unable to start apptime server in yarn -Ambari Hadoop

Expert Contributor

App timeline server us not able to connect to HDFS.

2018-03-29 15:30:44,338 ERROR
timeline.EntityGroupFSTimelineStore (EntityGroupFSTimelineStore.java:run(852))
- Error scanning active files
java.net.ConnectException: Call From hdpmaster.blpclean.com/192.168.1.10 to hdpmaster.blpclean.com:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.GeneratedConstructorAccessor217.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruct
2018-03-29 15:31:18,323 ERROR applicationhistoryservice.ApplicationHistoryServer (LogAdapter.java:error(69)) - RECEIVED SIGNAL 15: SIGTERM
2018-03-29 15:31:28,384 INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:8188

Please check whether HDFS services are up and check whether HDFS port 8020 is accessible from Timeline server.

Re: unable to start apptime server in yarn -Ambari Hadoop

Super Mentor

@shrinivas acharya


Please check the NameNode log, It looks like your NameNode is not running OR it might be showing some errors in the log. Or the NameNode host might be having some resource issues like "too many open files" or limited Memory availability? Because we see that the "App timeline host" is not able to connect to the NameNode. It might be due to a network issue as well. But better to check the NN logs to see if it has any issues and if the "" host can access the NN hsot & Port?

 2018-03-29 15:27:59,337 ERROR timeline.EntityGroupFSTimelineStore (EntityGroupFSTimelineStore.java:run(852)) - Error scanning active files
java.net.ConnectException: Call From hdpmaster.blpclean.com/192.168.1.10 to hdpmaster.xxxxx.com:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
    at sun.reflect.GeneratedConstructorAccessor217.newInstance(Unknown Source)


Check if from Apptimeline host if you can connect to your NN Host & port?

# telnet hdpmaster.xxxxx.com 8020   



On the NameNode hsot please check if the NameNode port 8020 is opened? If not then please try restarting the NameNodes and check the logs.

# netstat -tnlpa | grep 8020

.

Re: unable to start apptime server in yarn -Ambari Hadoop

Just now i restarted server itself and starting application still having issue a and name node and apptimeline in same server.