hiveserver2 and history server not running

[#1]

Hi,

i've upgraded ambari from 2.2 to 2.4 and hdp to 2.5

hiveserver2 and history server are not getting started

if i try to start history server from ambari i am getting the following error

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/historyserver.py", line 190, in <module>
    HistoryServer().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/historyserver.py", line 101, in start
    host_sys_prepped=params.host_sys_prepped)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/copy_tarball.py", line 257, in copy_to_hdfs
    replace_existing_files=replace_existing_files,
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 459, in action_create_on_execute
    self.action_delayed("create")
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 456, in action_delayed
    self.get_hdfs_resource_executor().action_delayed(action_name, self)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 255, in action_delayed
    self._create_resource()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 269, in _create_resource
    self._create_file(self.main_resource.resource.target, source=self.main_resource.resource.source, mode=self.mode)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 322, in _create_file
    self.util.run_command(target, 'CREATE', method='PUT', overwrite=True, assertable_result=False, file_to_put=source, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 179, in run_command
    _, out, err = get_user_call_output(cmd, user=self.run_user, logoutput=self.logoutput, quiet=False)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/get_user_call_output.py", line 61, in get_user_call_output
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X PUT --data-binary @/usr/hdp/2.5.3.0-37/hadoop/mapreduce.tar.gz 'http://nnode.aws.atlas.bdaas.com:50070/webhdfs/v1/hdp/apps/2.5.3.0-37/mapreduce/mapreduce.tar.gz?op=CREATE&user.name=hdfs&overwrite=True&permission=444' 1>/tmp/tmp0SGg9U 2>/tmp/tmpf9vt5e' returned 52. curl: (52) Empty reply from server
100

after starting this server, the namenode automatically gets stopped.

[#2]

@Sujatha Veeswar

this issue sounds similar to the following. See the accepted answer in the following link.

https://community.hortonworks.com/questions/63190/hdp243ambari-2410-hiveserver2-history-server-and-n...

copy the mapreduce.tar.gz as suggested in above link and see if this resolves it.

[#3]

@Sujatha Veeswar - it looks like some issue with your namenode as this webhdfs call doesn't work. Can you please verify if your HDFS is up and running fine? as suggested by @mqureshi, please also try to upload mapreduce.tar.gz manually and see if it helps.

[#4]

@Sujatha Veeswar

I have seen such issue when the NameNode runs OutOfMemory scenarios, In that case even if the webhdfs port remains opened but the service is not able to respond due to memory bottleneck hence we get empty response. So i will suggest to also check your NameNode Heap memory and if you find that it is reaching to certain limit then please increase the Xmx of NameNode.

[#5]

The issues has been reported by almost 5 users ( googled for "50070/webhdfs/v1/hdp/apps/2.5.3.0-37/mapreduce/mapreduce.tar.gz?op=CREATE") .The issue is due to default value set for "NameNode Java heap size" which is just 1 GB. Setting it to 4 GB in my system fixed the issue. There seems to be some issue in namenode which suddenly need heap of 4 GB when historyserver or hiveserver is started.