Issue in adding Hive service from ambari in HDP 2.5

[#1]

Hi Team,

We have set up a 3 node HDP 2.5 cluster using CentOS 6.5. When we are trying to add the Hive service from Ambari in that cluster we are receiving connection refused error (error happening in all the 3 nodes). However, we have successfully added other services without any issues. Please note that while adding service we are selecting New MySQL database, there is no existing MySQL database in that cluster. Need your help to address the issue. Error stack given below which came under "Hive Client Install" while adding the Hive service.

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_client.py", line 68, in <module>
    HiveClient().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_client.py", line 35, in install
    self.configure(env)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_client.py", line 43, in configure
    hive(name='client')
  File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py", line 282, in hive
    mode = 0644,
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 123, in action_create
    content = self._get_content()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 160, in _get_content
    return content()
  File "/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 51, in __call__
    return self.get_content()
  File "/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 193, in get_content
    web_file = opener.open(req)
  File "/usr/lib64/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 1190, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib64/python2.6/urllib2.py", line 1165, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 111] Connection refused>

[#2]

Please note that we are using local repo ......

[#3]

@Kuldeep Kulkarni - we are able to telnet and ping to ambari-server host from the reported host. Screenshots attached.

telnet-output.jpg

ping-output.jpg

[#4]

@rajdip chaudhuri - Sorry I thought dn1.tcsgegdc.com:8080 is your Ambari server. What is this machine, is this your repo server? Can you please check telnet for this machine?

[#5]

@Kuldeep Kulkarni - not able to do telnet from ambari server host to dn1.tcsgegdc.com. Can you please guide me what needs to be done here? Please note that telnet is installed in all the nodes in cluster.

[root@nn ambari-agent]# telnet dn1.tcsgegdc.com 8080 Trying 3.209.124.206... telnet: connect to address 3.209.124.206: Connection refused

[#6]

@Kuldeep Kulkarni - in dn1.tcsgedc.com we ZK service running and we are able to telnet on port 2181 from ambari server host. To give you a bit of synopsis:

nn.tcsgegdc.com -->> NN and Ambari server host

dn1.tcsgedc.com -->> Data Node and Hive host

dn2.tcsgedc.com -->> Data Node and Spark host

[#7 · Accepted solution]

@rajdip chaudhuri

It should download DBConnectionVerification.jar from Ambari server. I just double checked. Not sure why it is picking up different host instead of Ambari server. Can you please check your /etc/hosts file just to see if there is any conflicting entry.

This problem is happening while installing hive-client on your system.

Can you please make sure to have DBConnectionVerification.jar on all the ambari-agents?

/usr/lib/ambari-agent/DBConnectionVerification.jar
/var/lib/ambari-agent/tmp/DBConnectionVerification.jar

Typically, on Ambari server, below is the location for DBConnectionVerification.jar

/var/lib/ambari-server/resources/DBConnectionVerification.jar

[#8]

@Kuldeep Kulkarni - your solution of copying the jar file worked. Is this some kind of bug in HDP 2.5 because we did not faced this in 2.4 and 2.3.

[#9]

@rajdip chaudhuri - That's great. Can you please accept my answer above. I will look into this and file a BUG if needed.