Fail to start hiveserver2 and history server and also datanodes are starting and goinh down .

[#1]

stderr: /var/lib/ambari-agent/data/errors-538.txt

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py", line 211, in <module>
    HiveServer().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py", line 85, in start
    self.configure(env) # FOR SECURITY
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py", line 56, in configure
    hive(name='hiveserver2')
  File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py", line 140, in hive
    copy_to_hdfs("mapreduce", params.user_group, params.hdfs_user, host_sys_prepped=params.host_sys_prepped)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/copy_tarball.py", line 257, in copy_to_hdfs
    replace_existing_files=replace_existing_files,
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 459, in action_create_on_execute
    self.action_delayed("create")
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 456, in action_delayed
    self.get_hdfs_resource_executor().action_delayed(action_name, self)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 255, in action_delayed
    self._create_resource()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 269, in _create_resource
    self._create_file(self.main_resource.resource.target, source=self.main_resource.resource.source, mode=self.mode)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 322, in _create_file
    self.util.run_command(target, 'CREATE', method='PUT', overwrite=True, assertable_result=False, file_to_put=source, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 192, in run_command
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X PUT --data-binary @/usr/hdp/2.4.3.0-227/hadoop/mapreduce.tar.gz -H 'Content-Type: application/octet-stream' 'http://testnode01.localhost.com:50070/webhdfs/v1/hdp/apps/2.4.3.0-227/mapreduce/mapreduce.tar.gz?op=CREATE&user.name=hdfs&overwrite=True&permission=444'' returned status_code=403. 
{
  "RemoteException": {
    "exception": "IOException", 
    "javaClassName": "java.io.IOException", 
    "message": "Failed to find datanode, suggest to check cluster health."
  }
}

stdout: /var/lib/ambari-agent/data/output-538.txt

2017-07-06 04:07:36,606 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.4.3.0-227
2017-07-06 04:07:36,606 - Checking if need to create versioned conf dir /etc/hadoop/2.4.3.0-227/0
2017-07-06 04:07:36,606 - call[('ambari-python-wrap', '/usr/bin/conf-select', 'create-conf-dir', '--package', 'hadoop', '--stack-version', '2.4.3.0-227', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2017-07-06 04:07:36,628 - call returned (1, '/etc/hadoop/2.4.3.0-227/0 exist already', '')
2017-07-06 04:07:36,628 - checked_call[('ambari-python-wrap', '/usr/bin/conf-select', 'set-conf-dir', '--package', 'hadoop', '--stack-version', '2.4.3.0-227', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False}
2017-07-06 04:07:36,649 - checked_call returned (0, '')
2017-07-06 04:07:36,650 - Ensuring that hadoop has the correct symlink structure
2017-07-06 04:07:36,650 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-07-06 04:07:36,744 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.4.3.0-227
2017-07-06 04:07:36,745 - Checking if need to create versioned conf dir /etc/hadoop/2.4.3.0-227/0
2017-07-06 04:07:36,745 - call[('ambari-python-wrap', '/usr/bin/conf-select', 'create-conf-dir', '--package', 'hadoop', '--stack-version', '2.4.3.0-227', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2017-07-06 04:07:36,767 - call returned (1, '/etc/hadoop/2.4.3.0-227/0 exist already', '')
2017-07-06 04:07:36,767 - checked_call[('ambari-python-wrap', '/usr/bin/conf-select', 'set-conf-dir', '--package', 'hadoop', '--stack-version', '2.4.3.0-227', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False}
2017-07-06 04:07:36,784 - checked_call returned (0, '')
2017-07-06 04:07:36,784 - Ensuring that hadoop has the correct symlink structure
2017-07-06 04:07:36,785 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-07-06 04:07:36,786 - Group['ranger'] {}
2017-07-06 04:07:36,788 - Group['hadoop'] {}
2017-07-06 04:07:36,788 - Group['users'] {}
2017-07-06 04:07:36,788 - Group['knox'] {}
2017-07-06 04:07:36,788 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-07-06 04:07:36,789 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-07-06 04:07:36,789 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-07-06 04:07:36,789 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']}
2017-07-06 04:07:36,790 - User['ranger'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['ranger']}
2017-07-06 04:07:36,790 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']}
2017-07-06 04:07:36,791 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-07-06 04:07:36,791 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-07-06 04:07:36,791 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-07-06 04:07:36,792 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-07-06 04:07:36,792 - User['knox'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-07-06 04:07:36,793 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-07-06 04:07:36,794 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2017-07-06 04:07:36,799 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2017-07-06 04:07:36,800 - Group['hdfs'] {}
2017-07-06 04:07:36,800 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'hdfs']}
2017-07-06 04:07:36,801 - FS Type: 
2017-07-06 04:07:36,802 - Directory['/etc/hadoop'] {'mode': 0755}
2017-07-06 04:07:36,811 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2017-07-06 04:07:36,812 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2017-07-06 04:07:36,827 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2017-07-06 04:07:36,832 - Skipping Execute[('setenforce', '0')] due to not_if
2017-07-06 04:07:36,832 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
2017-07-06 04:07:36,835 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
2017-07-06 04:07:36,836 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}
2017-07-06 04:07:36,839 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2017-07-06 04:07:36,842 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2017-07-06 04:07:36,843 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': ..., 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2017-07-06 04:07:36,855 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs', 'group': 'hadoop'}
2017-07-06 04:07:36,856 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2017-07-06 04:07:36,857 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2017-07-06 04:07:36,860 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'}
2017-07-06 04:07:36,863 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2017-07-06 04:07:37,076 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.4.3.0-227
2017-07-06 04:07:37,076 - Checking if need to create versioned conf dir /etc/hadoop/2.4.3.0-227/0
2017-07-06 04:07:37,077 - call[('ambari-python-wrap', '/usr/bin/conf-select', 'create-conf-dir', '--package', 'hadoop', '--stack-version', '2.4.3.0-227', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2017-07-06 04:07:37,106 - call returned (1, '/etc/hadoop/2.4.3.0-227/0 exist already', '')
2017-07-06 04:07:37,107 - checked_call[('ambari-python-wrap', '/usr/bin/conf-select', 'set-conf-dir', '--package', 'hadoop', '--stack-version', '2.4.3.0-227', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False}
2017-07-06 04:07:37,133 - checked_call returned (0, '')
2017-07-06 04:07:37,134 - Ensuring that hadoop has the correct symlink structure
2017-07-06 04:07:37,134 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-07-06 04:07:37,141 - call['ambari-python-wrap /usr/bin/hdp-select status hive-server2'] {'timeout': 20}
2017-07-06 04:07:37,166 - call returned (0, 'hive-server2 - 2.4.3.0-227')
2017-07-06 04:07:37,167 - Stack Feature Version Info: stack_version=2.4, version=2.4.3.0-227, current_cluster_version=2.4.3.0-227 -> 2.4.3.0-227
2017-07-06 04:07:37,177 - HdfsResource['/user/hcat'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'dfs_type': '', 'default_fs': 'hdfs://testnode01.localhost.com:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': 'missing_principal', 'user': 'hdfs', 'owner': 'hcat', 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/apps/hive/warehouse', u'/mr-history/done', u'/app-logs', u'/tmp'], 'mode': 0755}
2017-07-06 04:07:37,182 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://testnode01.localhost.com:50070/webhdfs/v1/user/hcat?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmplluxRc 2>/tmp/tmppB72VE''] {'logoutput': None, 'quiet': False}
2017-07-06 04:07:37,212 - call returned (0, '')
2017-07-06 04:07:37,214 - Called copy_to_hdfs tarball: mapreduce
2017-07-06 04:07:37,215 - Default version is 2.4.3.0-227
2017-07-06 04:07:37,215 - Source file: /usr/hdp/2.4.3.0-227/hadoop/mapreduce.tar.gz , Dest file in HDFS: /hdp/apps/2.4.3.0-227/mapreduce/mapreduce.tar.gz
2017-07-06 04:07:37,215 - HdfsResource['/hdp/apps/2.4.3.0-227/mapreduce'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'dfs_type': '', 'default_fs': 'hdfs://testnode01.localhost.com:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': 'missing_principal', 'user': 'hdfs', 'owner': 'hdfs', 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/apps/hive/warehouse', u'/mr-history/done', u'/app-logs', u'/tmp'], 'mode': 0555}
2017-07-06 04:07:37,216 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://testnode01.localhost.com:50070/webhdfs/v1/hdp/apps/2.4.3.0-227/mapreduce?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpZmJokv 2>/tmp/tmps1eh6d''] {'logoutput': None, 'quiet': False}
2017-07-06 04:07:37,240 - call returned (0, '')
2017-07-06 04:07:37,241 - HdfsResource['/hdp/apps/2.4.3.0-227/mapreduce/mapreduce.tar.gz'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'source': '/usr/hdp/2.4.3.0-227/hadoop/mapreduce.tar.gz', 'dfs_type': '', 'default_fs': 'hdfs://testnode01.localhost.com:8020', 'replace_existing_files': False, 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': 'missing_principal', 'user': 'hdfs', 'owner': 'hdfs', 'group': 'hadoop', 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'file', 'action': ['create_on_execute'], 'immutable_paths': [u'/apps/hive/warehouse', u'/mr-history/done', u'/app-logs', u'/tmp'], 'mode': 0444}
2017-07-06 04:07:37,242 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://testnode01.localhost.com:50070/webhdfs/v1/hdp/apps/2.4.3.0-227/mapreduce/mapreduce.tar.gz?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpHOCC7J 2>/tmp/tmpq78pwX''] {'logoutput': None, 'quiet': False}
2017-07-06 04:07:37,267 - call returned (0, '')
2017-07-06 04:07:37,267 - Creating new file /hdp/apps/2.4.3.0-227/mapreduce/mapreduce.tar.gz in DFS
2017-07-06 04:07:37,269 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X PUT --data-binary @/usr/hdp/2.4.3.0-227/hadoop/mapreduce.tar.gz -H '"'"'Content-Type: application/octet-stream'"'"' '"'"'http://testnode01.localhost.com:50070/webhdfs/v1/hdp/apps/2.4.3.0-227/mapreduce/mapreduce.tar.gz?op=CREATE&user.name=hdfs&overwrite=True&permission=444'"'"' 1>/tmp/tmpf8_3BF 2>/tmp/tmpT4YGOg''] {'logoutput': None, 'quiet': False}
2017-07-06 04:07:38,631 - call returned (0, '')
 

Command failed after 1 tries

[#2]

@Abhishek Kumar

Please copy hiveserver2.log, history server log, datanode logs etc for understanding the issue and to assist you further on this.

Meantime, do check these log files yourself, might be meory related issue (check for 'outofmemory' text in these files), could be folder permission issue and so on.

For better targeted answer, need the log files.

[#3]

@Abhishek Kumar

Usually we see the following error when the DataNodes are not having the correct FQDN OR if the DataNodes are not running.

"message":"Failed to find datanode, suggest to check cluster health."

.

Can you please check your cluster Nodes (Specially all the DataNodes) are having the correct FQDN mapped properly in the configuration files ?

Please check and verify the cluster nodes FQDN: https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-installation/content/collect_info...

.