Created 01-02-2017 05:07 AM
Hi,
i've upgraded ambari from 2.2 to 2.4 and hdp to 2.5
hiveserver2 and history server are not getting started
if i try to start history server from ambari i am getting the following error
Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/historyserver.py", line 190, in <module> HistoryServer().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/historyserver.py", line 101, in start host_sys_prepped=params.host_sys_prepped) File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/copy_tarball.py", line 257, in copy_to_hdfs replace_existing_files=replace_existing_files, File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 459, in action_create_on_execute self.action_delayed("create") File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 456, in action_delayed self.get_hdfs_resource_executor().action_delayed(action_name, self) File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 255, in action_delayed self._create_resource() File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 269, in _create_resource self._create_file(self.main_resource.resource.target, source=self.main_resource.resource.source, mode=self.mode) File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 322, in _create_file self.util.run_command(target, 'CREATE', method='PUT', overwrite=True, assertable_result=False, file_to_put=source, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 179, in run_command _, out, err = get_user_call_output(cmd, user=self.run_user, logoutput=self.logoutput, quiet=False) File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/get_user_call_output.py", line 61, in get_user_call_output raise Fail(err_msg) resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X PUT --data-binary @/usr/hdp/2.5.3.0-37/hadoop/mapreduce.tar.gz 'http://nnode.aws.atlas.bdaas.com:50070/webhdfs/v1/hdp/apps/2.5.3.0-37/mapreduce/mapreduce.tar.gz?op=CREATE&user.name=hdfs&overwrite=True&permission=444' 1>/tmp/tmp0SGg9U 2>/tmp/tmpf9vt5e' returned 52. curl: (52) Empty reply from server 100
after starting this server, the namenode automatically gets stopped.
Created 01-02-2017 07:00 AM
this issue sounds similar to the following. See the accepted answer in the following link.
copy the mapreduce.tar.gz as suggested in above link and see if this resolves it.
Created 01-02-2017 09:44 AM
@Sujatha Veeswar - it looks like some issue with your namenode as this webhdfs call doesn't work. Can you please verify if your HDFS is up and running fine? as suggested by @mqureshi, please also try to upload mapreduce.tar.gz manually and see if it helps.
Created 01-02-2017 10:24 AM
I have seen such issue when the NameNode runs OutOfMemory scenarios, In that case even if the webhdfs port remains opened but the service is not able to respond due to memory bottleneck hence we get empty response. So i will suggest to also check your NameNode Heap memory and if you find that it is reaching to certain limit then please increase the Xmx of NameNode.
Created 01-02-2017 10:17 PM
The issues has been reported by almost 5 users ( googled for "50070/webhdfs/v1/hdp/apps/2.5.3.0-37/mapreduce/mapreduce.tar.gz?op=CREATE") .The issue is due to default value set for "NameNode Java heap size" which is just 1 GB. Setting it to 4 GB in my system fixed the issue. There seems to be some issue in namenode which suddenly need heap of 4 GB when historyserver or hiveserver is started.