Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search

Fresh Cluster Startup History Server Fails HDP

nayeem_janmoham
New Contributor

Created ‎01-19-2018 02:33 PM

I just finished installing HDP 2.6.3.0-235 on a bare metal 4 node home lab running centos7,

When starting it fails when as my secondary name node tries to start the history server

Files view is unreachable

The passwordless ssh is working, the host files are mapped correctly with fqdn's.

I am aware that there is a problem with the port 50070 according to logs but have no idea how to fix it, any help would be appreciated.

The dump of error logs:

stderr: /var/lib/ambari-agent/data/errors-2196.txt

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/historyserver.py", line 134, in <module>
    HistoryServer().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/historyserver.py", line 101, in start
    skip=params.sysprep_skip_copy_tarballs_hdfs)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/copy_tarball.py", line 267, in copy_to_hdfs
    replace_existing_files=replace_existing_files,
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 555, in action_create_on_execute
    self.action_delayed("create")
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 552, in action_delayed
    self.get_hdfs_resource_executor().action_delayed(action_name, self)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 287, in action_delayed
    self._create_resource()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 303, in _create_resource
    self._create_file(self.main_resource.resource.target, source=self.main_resource.resource.source, mode=self.mode)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 418, in _create_file
    self.util.run_command(target, 'CREATE', method='PUT', overwrite=True, assertable_result=False, file_to_put=source, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 199, in run_command
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X PUT --data-binary @/usr/hdp/2.6.3.0-235/hadoop/mapreduce.tar.gz -H 'Content-Type: application/octet-stream' 'http://master.janus.com:50070/webhdfs/v1/hdp/apps/2.6.3.0-235/mapreduce/mapreduce.tar.gz?op=CREATE&user.name=hdfs&overwrite=True&permission=444'' returned status_code=403. 
{
  "RemoteException": {
    "exception": "IOException", 
    "javaClassName": "java.io.IOException", 
    "message": "Failed to find datanode, suggest to check cluster health. excludeDatanodes=null"
  }
}

stdout: /var/lib/ambari-agent/data/output-2196.txt

2018-01-18 21:03:42,920 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2018-01-18 21:03:43,187 - Stack Feature Version Info: stack_version=2.6, version=2.6.3.0-235, current_cluster_version=2.6.3.0-235 -> 2.6.3.0-235
2018-01-18 21:03:43,201 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
User Group mapping (user_group) is missing in the hostLevelParams
2018-01-18 21:03:43,203 - Group['livy'] {}
2018-01-18 21:03:43,204 - Group['spark'] {}
2018-01-18 21:03:43,204 - Group['zeppelin'] {}
2018-01-18 21:03:43,205 - Group['hadoop'] {}
2018-01-18 21:03:43,205 - Group['users'] {}
2018-01-18 21:03:43,205 - Group['knox'] {}
2018-01-18 21:03:43,205 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-01-18 21:03:43,206 - User['infra-solr'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-01-18 21:03:43,207 - User['atlas'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-01-18 21:03:43,208 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-01-18 21:03:43,209 - User['falcon'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2018-01-18 21:03:43,210 - User['accumulo'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-01-18 21:03:43,211 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-01-18 21:03:43,212 - User['flume'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-01-18 21:03:43,213 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-01-18 21:03:43,213 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-01-18 21:03:43,214 - User['storm'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-01-18 21:03:43,215 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-01-18 21:03:43,216 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2018-01-18 21:03:43,217 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2018-01-18 21:03:43,218 - User['zeppelin'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'zeppelin', u'hadoop']}
2018-01-18 21:03:43,219 - User['livy'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-01-18 21:03:43,220 - User['mahout'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-01-18 21:03:43,221 - User['druid'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-01-18 21:03:43,222 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2018-01-18 21:03:43,223 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-01-18 21:03:43,224 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-01-18 21:03:43,225 - User['sqoop'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-01-18 21:03:43,226 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-01-18 21:03:43,226 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-01-18 21:03:43,227 - User['knox'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-01-18 21:03:43,228 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2018-01-18 21:03:43,230 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2018-01-18 21:03:43,236 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2018-01-18 21:03:43,237 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'}
2018-01-18 21:03:43,239 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2018-01-18 21:03:43,240 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2018-01-18 21:03:43,247 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if
2018-01-18 21:03:43,247 - Group['hdfs'] {}
2018-01-18 21:03:43,248 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': [u'hadoop', u'hdfs']}
2018-01-18 21:03:43,249 - FS Type: 
2018-01-18 21:03:43,249 - Directory['/etc/hadoop'] {'mode': 0755}
2018-01-18 21:03:43,284 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2018-01-18 21:03:43,286 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2018-01-18 21:03:43,315 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2018-01-18 21:03:43,322 - Skipping Execute[('setenforce', '0')] due to not_if
2018-01-18 21:03:43,323 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
2018-01-18 21:03:43,327 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
2018-01-18 21:03:43,328 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}
2018-01-18 21:03:43,337 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2018-01-18 21:03:43,341 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2018-01-18 21:03:43,355 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2018-01-18 21:03:43,379 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2018-01-18 21:03:43,380 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2018-01-18 21:03:43,382 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2018-01-18 21:03:43,390 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'}
2018-01-18 21:03:43,395 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2018-01-18 21:03:43,776 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2018-01-18 21:03:43,778 - call['ambari-python-wrap /usr/bin/hdp-select status hadoop-yarn-resourcemanager'] {'timeout': 20}
2018-01-18 21:03:43,809 - call returned (0, 'hadoop-yarn-resourcemanager - 2.6.3.0-235')
2018-01-18 21:03:43,823 - Stack Feature Version Info: stack_version=2.6, version=2.6.3.0-235, current_cluster_version=2.6.3.0-235 -> 2.6.3.0-235
2018-01-18 21:03:43,877 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2018-01-18 21:03:43,900 - HdfsResource['/app-logs'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'dfs_type': '', 'default_fs': 'hdfs://master.janus.com:8020', 'user': 'hdfs', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': [EMPTY], 'recursive_chmod': True, 'owner': 'yarn', 'group': 'hadoop', 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/apps/hive/warehouse', u'/apps/falcon', u'/mr-history/done', u'/app-logs', u'/tmp'], 'mode': 0777}
2018-01-18 21:03:43,903 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://master.janus.com:50070/webhdfs/v1/app-logs?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpSTJWBw 2>/tmp/tmpfFXJqb''] {'logoutput': None, 'quiet': False}
2018-01-18 21:03:43,977 - call returned (0, '')
2018-01-18 21:03:43,978 - Skipping the operation for not managed DFS directory /app-logs since immutable_paths contains it.
2018-01-18 21:03:43,978 - HdfsResource['/tmp'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'dfs_type': '', 'default_fs': 'hdfs://master.janus.com:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'owner': 'hdfs', 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/apps/hive/warehouse', u'/apps/falcon', u'/mr-history/done', u'/app-logs', u'/tmp'], 'mode': 0777}
2018-01-18 21:03:43,980 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://master.janus.com:50070/webhdfs/v1/tmp?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmplV3gJH 2>/tmp/tmpOwuJGd''] {'logoutput': None, 'quiet': False}
2018-01-18 21:03:44,058 - call returned (0, '')
2018-01-18 21:03:44,059 - Skipping the operation for not managed DFS directory /tmp since immutable_paths contains it.
2018-01-18 21:03:44,060 - HdfsResource['/tmp/entity-file-history/active'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'dfs_type': '', 'default_fs': 'hdfs://master.janus.com:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'owner': 'yarn', 'group': 'hadoop', 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/apps/hive/warehouse', u'/apps/falcon', u'/mr-history/done', u'/app-logs', u'/tmp']}
2018-01-18 21:03:44,062 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://master.janus.com:50070/webhdfs/v1/tmp/entity-file-history/active?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmp2vF_ey 2>/tmp/tmpiB9MzW''] {'logoutput': None, 'quiet': False}
2018-01-18 21:03:44,142 - call returned (0, '')
2018-01-18 21:03:44,144 - HdfsResource['/mapred'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'dfs_type': '', 'default_fs': 'hdfs://master.janus.com:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'owner': 'mapred', 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/apps/hive/warehouse', u'/apps/falcon', u'/mr-history/done', u'/app-logs', u'/tmp']}
2018-01-18 21:03:44,145 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://master.janus.com:50070/webhdfs/v1/mapred?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmp1KWlbW 2>/tmp/tmpbEFxVo''] {'logoutput': None, 'quiet': False}
2018-01-18 21:03:44,228 - call returned (0, '')
2018-01-18 21:03:44,229 - HdfsResource['/mapred/system'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'dfs_type': '', 'default_fs': 'hdfs://master.janus.com:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'owner': 'hdfs', 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/apps/hive/warehouse', u'/apps/falcon', u'/mr-history/done', u'/app-logs', u'/tmp']}
2018-01-18 21:03:44,231 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://master.janus.com:50070/webhdfs/v1/mapred/system?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpIsl_d1 2>/tmp/tmpL5N5NZ''] {'logoutput': None, 'quiet': False}
2018-01-18 21:03:44,311 - call returned (0, '')
2018-01-18 21:03:44,313 - HdfsResource['/mr-history/done'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'dfs_type': '', 'default_fs': 'hdfs://master.janus.com:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'change_permissions_for_parents': True, 'owner': 'mapred', 'group': 'hadoop', 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/apps/hive/warehouse', u'/apps/falcon', u'/mr-history/done', u'/app-logs', u'/tmp'], 'mode': 0777}
2018-01-18 21:03:44,315 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://master.janus.com:50070/webhdfs/v1/mr-history/done?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpN70qj8 2>/tmp/tmpmvLi2G''] {'logoutput': None, 'quiet': False}
2018-01-18 21:03:44,398 - call returned (0, '')
2018-01-18 21:03:44,400 - Skipping the operation for not managed DFS directory /mr-history/done since immutable_paths contains it.
2018-01-18 21:03:44,400 - HdfsResource[None] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'dfs_type': '', 'default_fs': 'hdfs://master.janus.com:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'action': ['execute'], 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'immutable_paths': [u'/apps/hive/warehouse', u'/apps/falcon', u'/mr-history/done', u'/app-logs', u'/tmp']}
2018-01-18 21:03:44,401 - Directory['/hadoop/mapreduce/jhs'] {'owner': 'mapred', 'group': 'hadoop', 'create_parents': True, 'recursive_ownership': True, 'cd_access': 'a'}
2018-01-18 21:03:44,408 - Directory['/var/log/hadoop-yarn/nodemanager/recovery-state'] {'owner': 'yarn', 'group': 'hadoop', 'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2018-01-18 21:03:44,410 - Directory['/var/run/hadoop-yarn'] {'owner': 'yarn', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2018-01-18 21:03:44,411 - Directory['/var/run/hadoop-yarn/yarn'] {'owner': 'yarn', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2018-01-18 21:03:44,412 - Directory['/var/log/hadoop-yarn/yarn'] {'owner': 'yarn', 'group': 'hadoop', 'create_parents': True, 'cd_access': 'a'}
2018-01-18 21:03:44,413 - Directory['/var/run/hadoop-mapreduce'] {'owner': 'mapred', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2018-01-18 21:03:44,414 - Directory['/var/run/hadoop-mapreduce/mapred'] {'owner': 'mapred', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2018-01-18 21:03:44,415 - Directory['/var/log/hadoop-mapreduce'] {'owner': 'mapred', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2018-01-18 21:03:44,416 - Directory['/var/log/hadoop-mapreduce/mapred'] {'owner': 'mapred', 'group': 'hadoop', 'create_parents': True, 'cd_access': 'a'}
2018-01-18 21:03:44,418 - Directory['/var/log/hadoop-yarn'] {'owner': 'yarn', 'group': 'hadoop', 'ignore_failures': True, 'create_parents': True, 'cd_access': 'a'}
2018-01-18 21:03:44,419 - XmlConfig['core-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {u'final': {u'fs.defaultFS': u'true'}}, 'owner': 'hdfs', 'configurations': ...}
2018-01-18 21:03:44,439 - Generating config: /usr/hdp/current/hadoop-client/conf/core-site.xml
2018-01-18 21:03:44,440 - File['/usr/hdp/current/hadoop-client/conf/core-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2018-01-18 21:03:44,477 - XmlConfig['hdfs-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {u'final': {u'dfs.support.append': u'true', u'dfs.datanode.data.dir': u'true', u'dfs.namenode.http-address': u'true', u'dfs.namenode.name.dir': u'true', u'dfs.webhdfs.enabled': u'true', u'dfs.datanode.failed.volumes.tolerated': u'true'}}, 'owner': 'hdfs', 'configurations': ...}
2018-01-18 21:03:44,494 - Generating config: /usr/hdp/current/hadoop-client/conf/hdfs-site.xml
2018-01-18 21:03:44,494 - File['/usr/hdp/current/hadoop-client/conf/hdfs-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2018-01-18 21:03:44,560 - XmlConfig['mapred-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'yarn', 'configurations': ...}
2018-01-18 21:03:44,573 - Generating config: /usr/hdp/current/hadoop-client/conf/mapred-site.xml
2018-01-18 21:03:44,574 - File['/usr/hdp/current/hadoop-client/conf/mapred-site.xml'] {'owner': 'yarn', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2018-01-18 21:03:44,628 - Changing owner for /usr/hdp/current/hadoop-client/conf/mapred-site.xml from 1023 to yarn
2018-01-18 21:03:44,629 - XmlConfig['yarn-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'yarn', 'configurations': ...}
2018-01-18 21:03:44,642 - Generating config: /usr/hdp/current/hadoop-client/conf/yarn-site.xml
2018-01-18 21:03:44,642 - File['/usr/hdp/current/hadoop-client/conf/yarn-site.xml'] {'owner': 'yarn', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2018-01-18 21:03:44,773 - XmlConfig['capacity-scheduler.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'yarn', 'configurations': ...}
2018-01-18 21:03:44,785 - Generating config: /usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml
2018-01-18 21:03:44,786 - File['/usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml'] {'owner': 'yarn', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2018-01-18 21:03:44,802 - Changing owner for /usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml from 1020 to yarn
2018-01-18 21:03:44,807 - File['/etc/security/limits.d/yarn.conf'] {'content': Template('yarn.conf.j2'), 'mode': 0644}
2018-01-18 21:03:44,810 - File['/etc/security/limits.d/mapreduce.conf'] {'content': Template('mapreduce.conf.j2'), 'mode': 0644}
2018-01-18 21:03:44,819 - File['/usr/hdp/current/hadoop-client/conf/yarn-env.sh'] {'content': InlineTemplate(...), 'owner': 'yarn', 'group': 'hadoop', 'mode': 0755}
2018-01-18 21:03:44,820 - Writing File['/usr/hdp/current/hadoop-client/conf/yarn-env.sh'] because contents don't match
2018-01-18 21:03:44,821 - File['/usr/hdp/current/hadoop-yarn-client/bin/container-executor'] {'group': 'hadoop', 'mode': 02050}
2018-01-18 21:03:44,823 - File['/usr/hdp/current/hadoop-client/conf/container-executor.cfg'] {'content': Template('container-executor.cfg.j2'), 'group': 'hadoop', 'mode': 0644}
2018-01-18 21:03:44,824 - Directory['/cgroups_test/cpu'] {'group': 'hadoop', 'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2018-01-18 21:03:44,827 - File['/usr/hdp/current/hadoop-client/conf/mapred-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'mode': 0755}
2018-01-18 21:03:44,830 - File['/usr/hdp/current/hadoop-client/conf/taskcontroller.cfg'] {'content': Template('taskcontroller.cfg.j2'), 'owner': 'hdfs'}
2018-01-18 21:03:44,831 - XmlConfig['mapred-site.xml'] {'owner': 'mapred', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2018-01-18 21:03:44,843 - Generating config: /usr/hdp/current/hadoop-client/conf/mapred-site.xml
2018-01-18 21:03:44,844 - File['/usr/hdp/current/hadoop-client/conf/mapred-site.xml'] {'owner': 'mapred', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2018-01-18 21:03:44,895 - Changing owner for /usr/hdp/current/hadoop-client/conf/mapred-site.xml from 1022 to mapred
2018-01-18 21:03:44,895 - XmlConfig['capacity-scheduler.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2018-01-18 21:03:44,908 - Generating config: /usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml
2018-01-18 21:03:44,908 - File['/usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2018-01-18 21:03:44,926 - Changing owner for /usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml from 1022 to hdfs
2018-01-18 21:03:44,926 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2018-01-18 21:03:44,939 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-client.xml
2018-01-18 21:03:44,939 - File['/usr/hdp/current/hadoop-client/conf/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2018-01-18 21:03:44,947 - Directory['/usr/hdp/current/hadoop-client/conf/secure'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2018-01-18 21:03:44,948 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf/secure', 'configuration_attributes': {}, 'configurations': ...}
2018-01-18 21:03:44,961 - Generating config: /usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml
2018-01-18 21:03:44,961 - File['/usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2018-01-18 21:03:44,969 - XmlConfig['ssl-server.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...}
2018-01-18 21:03:44,982 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-server.xml
2018-01-18 21:03:44,982 - File['/usr/hdp/current/hadoop-client/conf/ssl-server.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2018-01-18 21:03:44,992 - File['/usr/hdp/current/hadoop-client/conf/ssl-client.xml.example'] {'owner': 'mapred', 'group': 'hadoop'}
2018-01-18 21:03:44,992 - File['/usr/hdp/current/hadoop-client/conf/ssl-server.xml.example'] {'owner': 'mapred', 'group': 'hadoop'}
2018-01-18 21:03:44,997 - Called copy_to_hdfs tarball: mapreduce
2018-01-18 21:03:44,997 - Default version is 2.6.3.0-235
2018-01-18 21:03:44,997 - Source file: /usr/hdp/2.6.3.0-235/hadoop/mapreduce.tar.gz , Dest file in HDFS: /hdp/apps/2.6.3.0-235/mapreduce/mapreduce.tar.gz
2018-01-18 21:03:44,998 - HdfsResource['/hdp/apps/2.6.3.0-235/mapreduce'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'dfs_type': '', 'default_fs': 'hdfs://master.janus.com:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'owner': 'hdfs', 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/apps/hive/warehouse', u'/apps/falcon', u'/mr-history/done', u'/app-logs', u'/tmp'], 'mode': 0555}
2018-01-18 21:03:44,999 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://master.janus.com:50070/webhdfs/v1/hdp/apps/2.6.3.0-235/mapreduce?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmp7PhsKQ 2>/tmp/tmpNlYQ3U''] {'logoutput': None, 'quiet': False}
2018-01-18 21:03:45,076 - call returned (0, '')
2018-01-18 21:03:45,077 - HdfsResource['/hdp/apps/2.6.3.0-235/mapreduce/mapreduce.tar.gz'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'source': '/usr/hdp/2.6.3.0-235/hadoop/mapreduce.tar.gz', 'dfs_type': '', 'default_fs': 'hdfs://master.janus.com:8020', 'replace_existing_files': False, 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': 'kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'owner': 'hdfs', 'group': 'hadoop', 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'file', 'action': ['create_on_execute'], 'immutable_paths': [u'/apps/hive/warehouse', u'/apps/falcon', u'/mr-history/done', u'/app-logs', u'/tmp'], 'mode': 0444}
2018-01-18 21:03:45,079 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://master.janus.com:50070/webhdfs/v1/hdp/apps/2.6.3.0-235/mapreduce/mapreduce.tar.gz?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmp2BWebj 2>/tmp/tmpDPVO0z''] {'logoutput': None, 'quiet': False}
2018-01-18 21:03:45,163 - call returned (0, '')
2018-01-18 21:03:45,164 - Creating new file /hdp/apps/2.6.3.0-235/mapreduce/mapreduce.tar.gz in DFS
2018-01-18 21:03:45,165 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X PUT --data-binary @/usr/hdp/2.6.3.0-235/hadoop/mapreduce.tar.gz -H '"'"'Content-Type: application/octet-stream'"'"' '"'"'http://master.janus.com:50070/webhdfs/v1/hdp/apps/2.6.3.0-235/mapreduce/mapreduce.tar.gz?op=CREATE&user.name=hdfs&overwrite=True&permission=444'"'"' 1>/tmp/tmptEcbeB 2>/tmp/tmpLx6LWX''] {'logoutput': None, 'quiet': False}
2018-01-18 21:03:45,365 - call returned (0, '')

Command failed after 1 tries
992 Views
0 Kudos
1 REPLY 1

yvora
Guru

Created ‎01-20-2018 09:16 PM

@Nayeem Janmohamed, It seems that datanodes in this cluster are not in good state. Can you please restart datanodes and restart History Server ?

596 Views
Take a Tour of the Community
Don't have an account?
Register
Your experience may be limited. Sign in to explore more.
Announcements
Community Announcements
March 2023 Community Highlights
What's New @ Cloudera
Cloudera DataFlow Designer for self-service data flow development is now generally available to all CDP Public Cloud customers
What's New @ Cloudera
Cloudera Operational Database (COD) UI provides the JWT configuration details to connect to your HBase client
What's New @ Cloudera
Cloudera Operational Database (COD) UI allows storage type selection when creating an operational database
What's New @ Cloudera
Release of Cloudera Streaming Analytics (CSA) 1.9.0 for CDP 7.1.7 SP2 & CDP 7.1.8
View More Announcements