HDP2.4.3::Ambari 2.4.1.0 ::HiveServer2, History Server and NodeManager Not Starting

[#1]

Attached Out Logs: hiveserver2err.txt

history-server-hdp243err.txt

HDP2.4.3::Ambari 2.4.1.0 ::HiveServer2, History Server and NodeManager Not Starting

I tried

-- Restarting CLuster

-- Safe Mode OFF

NameNode starts - stays up for some time, but silently goes down.

HiveServer2 - Never Started

HistoryServer: Never Started

Rest All - Started and Stayed up.

When I attempt to Start HiveServer2 or HistoryServer/MR - it will bring "NameNode" down.

History Server:

File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 179, in run_command
    _, out, err = get_user_call_output(cmd, user=self.run_user, logoutput=self.logoutput, quiet=False)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/get_user_call_output.py", line 61, in get_user_call_output
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X PUT --data-binary @/usr/hdp/2.4.3.0-227/hadoop/mapreduce.tar.gz 'http://node09.example.com:50070/webhdfs/v1/hdp/apps/2.4.3.0-227/mapreduce/mapreduce.tar.gz?op=CREATE&user.name=hdfs&overwrite=True&permission=444' 1>/tmp/tmp4_0mSH 2>/tmp/tmpJCqPnb' returned 52. curl: (52) Empty reply from server 

100

HiveServer2:

  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/get_user_call_output.py", line 61, in get_user_call_output
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X PUT --data-binary @/usr/hdp/2.4.3.0-227/hadoop/mapreduce.tar.gz 'http://node09.example.com:50070/webhdfs/v1/hdp/apps/2.4.3.0-227/mapreduce/mapreduce.tar.gz?op=CREATE&user.name=hdfs&overwrite=True&permission=444' 1>/tmp/tmp9oruFg 2>/tmp/tmp6NevU5' returned 52. curl: (52) Empty reply from server 

100

[#2]

@sun pepper please paste the ambari agent logs.

[#3]

ambari-agentlog.txt

Sequence: Cleared Previous Agent Logs --> Ambari Login -> Nodes -> Node manager -> Start

Waited for few mins

Node Manager went down automatic.

[#4]

#2: Sequence: Truncate Ambari Server and Agent logs

Start NodeManager, Then start HiveServer2

Logs Attached.

ambari-serverloghive.txt

ambari-agentloghive.txt

[#5]

ERROR 2016-10-26 08:09:38,619 script_alert.py:119 - [Alert][yarn_nodemanager_health] Failed with result CRITICAL: ['Connection failed to http://xxxxx.example.com:8042


Can you make sure firewall ports are open to this machine and network connectivity is functional.

[#6]

@sun pepper port 8042 belongs to YARN nodemanager API. Please see this https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_installing_manually_book/content/configu...

also make sure ip4firewall is off as well as SELinux. If all else fails try using telnet to that port to make sure it is open.

[#7]

Please see my Firewall and other info below. No luck still. Cannot get my NodeManager, Hive UP. Those are must haves for me.

[#8]

@Artem Ervits may I know which service runs on 8042? I looked at this to see which one it is. can't fine 8042.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_HDP_Reference_Guide/content/accumulo-por...

ip6tables: was running. I just turned if off now.

iptables: was off since couple of days.

[root@~]# service ip6tables status

ip6tables: Firewall is not running.

[root@~]# service iptables status

iptables: Firewall is not running.

Tried Restarting NodeManager. Still Same

ERROR 2016-10-26 09:45:38,614 script_alert.py:119 - [Alert][yarn_nodemanager_health] Failed with result CRITICAL: ['Connection failed to http://node09.example.com:8042/ws/v1/node/info (Traceback (most recent call last):\n File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/alerts/alert_nodemanager_health.py", line 171, in execute\n url_response = urllib2.urlopen(query, timeout=connection_timeout)\n File "/usr/lib64/python2.6/urllib2.py", line 126, in urlopen\n return _opener.open(url, data, timeout)\n File "/usr/lib64/python2.6/urllib2.py", line 391, in open\n response = self._open(req, data)\n File "/usr/lib64/python2.6/urllib2.py", line 409, in _open\n \'_open\', req)\n File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain\n result = func(*args)\n File "/usr/lib64/python2.6/urllib2.py", line 1190, in http_open\n return self.do_open(httplib.HTTPConnection, req)\n File "/usr/lib64/python2.6/urllib2.py", line 1165, in do_open\n raise URLError(err)\nURLError: <urlopen error [Errno 111] Connection refused>\n)']

[#9]

One more Puzzle. SafeMode turns ON automatic some times.

hdfs dfsadmin -fs hdfs://node09.example.com:8020 -safemode get

Safe mode is ON

I manually turned off:

[hdfs@node09 ~]$ hdfs dfsadmin -fs hdfs://node09.example.com:8020 -safemode leave

Then:

-- Restarted NameNode

-- Started NodeManager: Still turns off after some time

-- Started HiveServer2: Still doesnt come up. Never did so far.

[#10 · Accepted solution]

*WORKED*

1: su - hdfs

hdfs dfs -put /usr/hdp/2.4.3.0-227/hadoop/mapreduce.tar.gz /hdp/apps/2.4.3.0-227/mapreduce/

3: su - atlas

cp /usr/hdp/2.4.3.0-227/etc/atlas/conf.dist/client.properties /etc/atlas/conf/