Support Questions
Find answers, ask questions, and share your expertise

History server and hive server2 is not coming up in HDP 2.3 using ambari 2.2.

History server and hive server2 is not coming up in HDP 2.3 using ambari 2.2.

Please find the error while bringing up history server:--

Traceback (most recent call last):

  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/historyserver.py", line 182, in <module>
    HistoryServer().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/historyserver.py", line 100, in start
    host_sys_prepped=params.host_sys_prepped)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/copy_tarball.py", line 201, in copy_to_hdfs
    replace_existing_files=replace_existing_files,
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 402, in action_create_on_execute
    self.action_delayed("create")
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 399, in action_delayed
    self.get_hdfs_resource_executor().action_delayed(action_name, self)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 255, in action_delayed
    self._create_resource()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 269, in _create_resource
    self._create_file(self.main_resource.resource.target, source=self.main_resource.resource.source, mode=self.mode)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 322, in _create_file
    self.util.run_command(target, 'CREATE', method='PUT', overwrite=True, assertable_result=False, file_to_put=source, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 210, in run_command
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X PUT -T /usr/hdp/2.3.4.0-3485/hadoop/mapreduce.tar.gz 'http://namenode:50070/webhdfs/v1/hdp/apps/2.3.4.0-3485/mapreduce/mapreduce.tar.gz?op=CREATE&user.name=hdfs&overwrite=True&permission=444'' returned status_code=403. 
{
  "RemoteException": {
    "exception": "IOException", 
    "javaClassName": "java.io.IOException", 
    "message": "Failed to find datanode, suggest to check cluster health."
  }
18 REPLIES 18
Highlighted

Re: History server and hive server2 is not coming up in HDP 2.3 using ambari 2.2.

Mentor
@shaheer mannan

doublecheck your hdfs is up, error says datanode is not found.

Highlighted

Re: History server and hive server2 is not coming up in HDP 2.3 using ambari 2.2.

Yes hdfs and yarn both are UP.

yarn.png

hdfs.png

ambari.png

Highlighted

Re: History server and hive server2 is not coming up in HDP 2.3 using ambari 2.2.

Mentor

@shaheer mannan run service checks for all services, specifically HDFS and YARN.

Highlighted

Re: History server and hive server2 is not coming up in HDP 2.3 using ambari 2.2.

New Contributor

I also have problems like @shaheer mannan. History server and hive server can't be started, hdfs and yarn both are ok, however when I run service checks for HDFS and YARN then an error occurs

HDFS Error:

1524-capture.jpg

YARN Error:

1525-capture.jpg

PLease help me! thank you so much

Highlighted

Re: History server and hive server2 is not coming up in HDP 2.3 using ambari 2.2.

Mentor

Please open a new thread as we did not identify solution here yet. @JetHL

Highlighted

Re: History server and hive server2 is not coming up in HDP 2.3 using ambari 2.2.

New Contributor

ok, tks Artem Ervits

Highlighted

Re: History server and hive server2 is not coming up in HDP 2.3 using ambari 2.2.

@shaheer mannan

Couple of things to check

http://namenode:50070/webhdfs/v1/hdp/apps/2.3.4.0-3485/mapreduce/mapreduce.tar.gz?op=CREATE&user.nam... "returned status_code=403."

in above: replace namenode with datanode

Wonderful tutorial http://hortonworks.com/blog/webhdfs-%E2%80%93-http-rest-access-to-hdfs/

A web server may return a 403 Forbidden HTTP status code in response to a request from a client for a web page or resource to indicate that the server can be reached and understood the request, but refuses to take any further action.

Highlighted

Re: History server and hive server2 is not coming up in HDP 2.3 using ambari 2.2.

Still it didn't work and I could see most of the people are facing this issue with HDP 2.3

Is it a bug or something.

Highlighted

Re: History server and hive server2 is not coming up in HDP 2.3 using ambari 2.2.

@shaheer mannan Please restart all the services and then share the details.