question Re: Issue in adding Hive service from ambari in HDP 2.5 in Archives of Support Questions (Read Only)

Issue in adding Hive service from ambari in HDP 2.5

rajdip_chaudhur — Mon, 31 Oct 2016 20:09:29 GMT

Hi Team,

We have set up a 3 node HDP 2.5 cluster using CentOS 6.5. When we are trying to add the Hive service from Ambari in that cluster we are receiving connection refused error (error happening in all the 3 nodes). However, we have successfully added other services without any issues. Please note that while adding service we are selecting New MySQL database, there is no existing MySQL database in that cluster. Need your help to address the issue. Error stack given below which came under "Hive Client Install" while adding the Hive service.

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_client.py", line 68, in <module>
    HiveClient().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_client.py", line 35, in install
    self.configure(env)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_client.py", line 43, in configure
    hive(name='client')
  File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py", line 282, in hive
    mode = 0644,
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 123, in action_create
    content = self._get_content()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 160, in _get_content
    return content()
  File "/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 51, in __call__
    return self.get_content()
  File "/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 193, in get_content
    web_file = opener.open(req)
  File "/usr/lib64/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 1190, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib64/python2.6/urllib2.py", line 1165, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 111] Connection refused>

Re: Issue in adding Hive service from ambari in HDP 2.5

rajdip_chaudhur — Mon, 31 Oct 2016 20:13:32 GMT

Please note that all the pre-requisites like SSH setup, THP disable, SELinux disable, iptables disable have been done in all the nodes in the cluster.

Re: Issue in adding Hive service from ambari in HDP 2.5

Arun- — Mon, 31 Oct 2016 20:16:24 GMT

Hi,

Please check ambari server log file and also verify the ports are open for MYSQL 3306 on that host.

Re: Issue in adding Hive service from ambari in HDP 2.5

rajdip_chaudhur — Mon, 31 Oct 2016 20:24:00 GMT

Hi,

Ambari server log snippet below and 3306 port is open.

31 Oct 2016 18:52:09,307 INFO [ambari-client-thread-23] AbstractProviderModule:953 - Current Metrics collector Host : null 31 Oct 2016 18:52:09,308 INFO [ambari-client-thread-23] AbstractProviderModule:958 - New Metrics collector Host : nn.tcsgegdc.com 31 Oct 2016 18:52:10,792 INFO [ambari-heartbeat-processor-0] ServiceComponentHostImpl:1067 - Host role transitioned to a new state, serviceComponentName=HCAT, hostName=dn2.tcsgegdc.com, oldState=INSTALLING, currentState=INSTALLED 31 Oct 2016 18:52:13,741 ERROR [ambari-heartbeat-processor-0] HeartbeatProcessor:554 - Operation failed - may be retried. Service component host: HIVE_CLIENT, host: dn2.tcsgegdc.com Action id 83-0 and Task id 384 31 Oct 2016 18:52:13,753 INFO [ambari-heartbeat-processor-0] ServiceComponentHostImpl:1067 - Host role transitioned to a new state, serviceComponentName=HIVE_CLIENT, hostName=dn2.tcsgegdc.com, oldState=INSTALLING, currentState=INSTALL_FAILED 31 Oct 2016 18:52:13,796 INFO [ambari-heartbeat-processor-0] ServiceComponentHostImpl:1067 - Host role transitioned to a new state, serviceComponentName=HIVE_METASTORE, hostName=dn2.tcsgegdc.com, oldState=INSTALLING, currentState=INSTALLED 31 Oct 2016 18:52:14,593 WARN [ambari-action-scheduler] ActionScheduler:415 - HIVE_CLIENT failed, request 83 will be aborted 31 Oct 2016 18:52:14,593 ERROR [ambari-action-scheduler] ActionScheduler:428 - Operation completely failed, aborting request id: 83 31 Oct 2016 18:52:14,606 INFO [ambari-action-scheduler] ServiceComponentHostImpl:1067 - Host role transitioned to a new state, serviceComponentName=HIVE_SERVER, hostName=dn2.tcsgegdc.com, oldState=INSTALLING, currentState=INSTALL_FAILED 31 Oct 2016 18:52:14,614 INFO [ambari-action-scheduler] ServiceComponentHostImpl:1067 - Host role transitioned to a new state, serviceComponentName=MYSQL_SERVER, hostName=dn2.tcsgegdc.com, oldState=INSTALLING, currentState=INSTALL_FAILED 31 Oct 2016 18:52:14,623 INFO [ambari-action-scheduler] ServiceComponentHostImpl:1067 - Host role transitioned to a new state, serviceComponentName=WEBHCAT_SERVER, hostName=dn2.tcsgegdc.com, oldState=INSTALLING, currentState=INSTALL_FAILED 31 Oct 2016 18:52:14,632 INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname dn2.tcsgegdc.com role HIVE_SERVER requestId null taskId 386 stageId null 31 Oct 2016 18:52:14,632 INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname dn2.tcsgegdc.com role MYSQL_SERVER requestId null taskId 387 stageId null 31 Oct 2016 18:52:14,632 INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname dn2.tcsgegdc.com role WEBHCAT_SERVER requestId null taskId 388 stageId null 31 Oct 2016 18:52:14,747 INFO [ambari-heartbeat-processor-0] HeartbeatProcessor:626 - Security of service component HCAT of service HIVE of cluster TCSGEINTERNALCLUSTER has changed from UNSECURED to UNKNOWN on host dn2.tcsgegdc.com

Re: Issue in adding Hive service from ambari in HDP 2.5

rajdip_chaudhur — Mon, 31 Oct 2016 20:32:54 GMT

adding the snapshot of ambari-alerts.log

ambari-alerts.jpg

Re: Issue in adding Hive service from ambari in HDP 2.5

KuldeepK — Mon, 31 Oct 2016 20:50:27 GMT

@rajdip chaudhuri

On which host you have installed MySQL server? Can you please double check if MySql is up and running? Can you also try to login to mysql shell from any other node in the cluster and see if its working?

Re: Issue in adding Hive service from ambari in HDP 2.5

rajdip_chaudhur — Mon, 31 Oct 2016 20:58:48 GMT

Hi @Kuldeep Kulkarni.....we are selecting new MySQL instance while adding the Hive service so MySQL is not pre-configured. We have deleted the service from Ambari as Hive services are having issues.

Re: Issue in adding Hive service from ambari in HDP 2.5

KuldeepK — Mon, 31 Oct 2016 21:01:49 GMT

@rajdip chaudhuri - okay. can you please share more logs when you get the error i.e. stdout and stderr from Ambari UI.

Re: Issue in adding Hive service from ambari in HDP 2.5

rajdip_chaudhur — Mon, 31 Oct 2016 21:07:26 GMT

@Kuldeep Kulkarni - below are logs from Ambari UI.

error-logs.txt

Re: Issue in adding Hive service from ambari in HDP 2.5

rajdip_chaudhur — Mon, 31 Oct 2016 21:11:01 GMT

Please note that we are using local repo ......

Re: Issue in adding Hive service from ambari in HDP 2.5

KuldeepK — Mon, 31 Oct 2016 21:11:46 GMT

@rajdip chaudhuri - Okay below command is failing:

Downloading the file from http://dn1.tcsgegdc.com:8080/resources/DBConnectionVerification.jar

Can you please make sure that you are able to connect to Ambari server from problematic host where above command is failing? you can try simple telnet to check the connectivity on port 8080

Re: Issue in adding Hive service from ambari in HDP 2.5

rajdip_chaudhur — Mon, 31 Oct 2016 21:20:25 GMT

@Kuldeep Kulkarni - we are able to telnet and ping to ambari-server host from the reported host. Screenshots attached.

telnet-output.jpg

ping-output.jpg

Re: Issue in adding Hive service from ambari in HDP 2.5

KuldeepK — Mon, 31 Oct 2016 21:22:58 GMT

@rajdip chaudhuri - Sorry I thought dn1.tcsgegdc.com:8080 is your Ambari server. What is this machine, is this your repo server? Can you please check telnet for this machine?

Re: Issue in adding Hive service from ambari in HDP 2.5

rajdip_chaudhur — Mon, 31 Oct 2016 21:32:15 GMT

@Kuldeep Kulkarni - not able to do telnet from ambari server host to dn1.tcsgegdc.com. Can you please guide me what needs to be done here? Please note that telnet is installed in all the nodes in cluster.

[root@nn ambari-agent]# telnet dn1.tcsgegdc.com 8080 Trying 3.209.124.206... telnet: connect to address 3.209.124.206: Connection refused

Re: Issue in adding Hive service from ambari in HDP 2.5

rajdip_chaudhur — Mon, 31 Oct 2016 21:43:37 GMT

@Kuldeep Kulkarni - in dn1.tcsgedc.com we ZK service running and we are able to telnet on port 2181 from ambari server host. To give you a bit of synopsis:

nn.tcsgegdc.com -->> NN and Ambari server host

dn1.tcsgedc.com -->> Data Node and Hive host

dn2.tcsgedc.com -->> Data Node and Spark host

Re: Issue in adding Hive service from ambari in HDP 2.5

KuldeepK — Mon, 31 Oct 2016 21:53:50 GMT

@rajdip chaudhuri

It should download DBConnectionVerification.jar from Ambari server. I just double checked. Not sure why it is picking up different host instead of Ambari server. Can you please check your /etc/hosts file just to see if there is any conflicting entry.

This problem is happening while installing hive-client on your system.

Can you please make sure to have DBConnectionVerification.jar on all the ambari-agents?

/usr/lib/ambari-agent/DBConnectionVerification.jar
/var/lib/ambari-agent/tmp/DBConnectionVerification.jar

Typically, on Ambari server, below is the location for DBConnectionVerification.jar

/var/lib/ambari-server/resources/DBConnectionVerification.jar

Re: Issue in adding Hive service from ambari in HDP 2.5

rajdip_chaudhur — Mon, 31 Oct 2016 22:08:41 GMT

@Kuldeep Kulkarni - your solution of copying the jar file worked. Is this some kind of bug in HDP 2.5 because we did not faced this in 2.4 and 2.3.

Re: Issue in adding Hive service from ambari in HDP 2.5

KuldeepK — Mon, 31 Oct 2016 22:11:43 GMT

@rajdip chaudhuri - That's great. Can you please accept my answer above. I will look into this and file a BUG if needed.