question Re: can't start hiveserver2 with ambari - beeline error Error: Could not open client transport with JDBC Uri in Support Questions

can't start hiveserver2 with ambari - beeline error Error: Could not open client transport with JDBC Uri

neomerdien — Sun, 22 May 2016 21:09:36 GMT

Hi,

when i try to start hiveserver2 using ambari i get the following error:

Connection failed on host HOSTNAME:10000 (Execution of '! beeline -u 'jdbc:hive2://HOSTNAME:10000/;transportMode=binary' -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'' returned 1.

Error: Could not open client transport with JDBC Uri: jdbc:hive2://HOSTNAME:10000/; transportMode=binary: java.net.ConnectException: Connection refused (state=08S01,code=0)

Error: Could not open client transport with JDBC Uri: jdbc:hive2://HOSTNAME:10000/; transportMode=binary: java.net.ConnectException: Connection refused (state=08S01,code=0))

i cant connect locally using: beeline -u 'jdbc:hive2://

even though the service is down.

hive metastore and WebHCat server are running.

I understand that the reason i cant connect is because hiveserver2 is down, but interestingly enough hive-server2.log (/var/log/hive) remains blank when i start the service so I cant understand what it fails to start in the first place.

HDP version 2.3.2.0-2950

Ambari version 2.1.2

any help will be appreciated.

Re: can't start hiveserver2 with ambari - beeline error Error: Could not open client transport with JDBC Uri

sshimpi — Sun, 22 May 2016 21:17:57 GMT

@Andrey Nikitin

Enable hive debug and check if you are able to see logs.

Also once hive is up try with below commands -

# beeline

>!connect jdbc:hive2://HOSTNAME:10000/default

Else try -

>!connect jdbc:hive2://<host>:<port>/<db>;transportMode=http;httpPath=<http_endpoint>

where:-

<http_endpoint> is the corresponding HTTP endpoint configured in hive-site.xml. Default value is cliservice.

Default port for HTTP transport mode is 10001.

If you are having kerberized cluster then user -

>!connect jdbc:hive2://<host>:10000/;principal=<Server_Principal_of_HiveServer2>

Re: can't start hiveserver2 with ambari - beeline error Error: Could not open client transport with JDBC Uri

neomerdien — Sun, 22 May 2016 21:34:23 GMT

Hi @Sagar Shimpi

how do i enable Hive debug?

Thanks.

Re: can't start hiveserver2 with ambari - beeline error Error: Could not open client transport with JDBC Uri

KuldeepK — Sun, 22 May 2016 22:06:05 GMT

@Andrey Nikitin

In addition to what @Sagar Shimpi said, can you please run below command on hiveserver2 host to see DEBUG output on console? it should give you a hint

Please run below command as hive user:

/usr/bin/hive --service hiveserver2 --hiveconf hive.root.logger=DEBUG,console

Re: can't start hiveserver2 with ambari - beeline error Error: Could not open client transport with JDBC Uri

neomerdien — Mon, 23 May 2016 14:47:18 GMT

Hi @Sagar Shimpi , @Kuldeep Kulkarni

some updates:

I've started hiveserver2 like you mentioned: /usr/bin/hive --service hiveserver2 --hiveconf hive.root.logger=DEBUG,console and it started without any issues and there was no apparent errors in the log, only these two warnings but i dont think its relevant: WARNING: Use "yarn jar" to launch YARN applications. 16/05/23 10:41:17 WARN conf.HiveConf: HiveConf of name hive.heapsize does not exist

after the service was started via the above command, I tried to "start" the service in ambari and it connected succesfully, also via the CLI with the following command: beeline -u 'jdbc:hive2://HOSTNAME:10000/;transportMode=binary'

when I tried to stop the service from ambari, it showed me that the services was stopped but when I checked the server it was still running

all of the above leads me to the conclusion that there is some problem between ambari and hiveserver2, its like its unable to start/stop the hiveserver2 service, and that is probably the reason why the log files are not update

please advise on what could an issue between ambari and running the services? i dont have problems with other services running from ambari on the same server.

Thanks.

Re: can't start hiveserver2 with ambari - beeline error Error: Could not open client transport with JDBC Uri

sshimpi — Mon, 23 May 2016 17:04:28 GMT

@Andrey Nikitin

Just give a try of restarting ambari server and agent(on which hiveserver2 runs)

Also pls let us know the version of ambari and hdp you are using.

Re: can't start hiveserver2 with ambari - beeline error Error: Could not open client transport with JDBC Uri

neomerdien — Mon, 23 May 2016 17:44:48 GMT

@Sagar Shimpi

i restarted the ambari server and agent on the hive machine and the outcome is the same.

this is the log from ambari-agent:

INFO 2016-05-23 13:42:52,555 RecoveryManager.py:217 - HIVE_SERVER needs recovery.
INFO 2016-05-23 13:42:54,316 RecoveryManager.py:217 - HIVE_SERVER needs recovery.
INFO 2016-05-23 13:43:02,556 Heartbeat.py:78 - Building Heartbeat: {responseId = 29, timestamp = 1464000182555, commandsInProgress = False, componentsMapped = True}
INFO 2016-05-23 13:43:02,577 Controller.py:260 - Heartbeat response received (id = 30)
INFO 2016-05-23 13:43:02,577 RecoveryManager.py:217 - HIVE_SERVER needs recovery.
INFO 2016-05-23 13:43:12,578 Heartbeat.py:78 - Building Heartbeat: {responseId = 30, timestamp = 1464000192578, commandsInProgress = False, componentsMapped = True}
INFO 2016-05-23 13:43:12,779 Controller.py:260 - Heartbeat response received (id = 31)
INFO 2016-05-23 13:43:12,780 RecoveryManager.py:217 - HIVE_SERVER needs recovery.
INFO 2016-05-23 13:43:22,781 Heartbeat.py:78 - Building Heartbeat: {responseId = 31, timestamp = 1464000202781, commandsInProgress = False, componentsMapped = True}
INFO 2016-05-23 13:43:22,825 Controller.py:260 - Heartbeat response received (id = 32)
INFO 2016-05-23 13:43:22,826 RecoveryManager.py:217 - HIVE_SERVER needs recovery.
WARNING 2016-05-23 13:43:29,367 base_alert.py:417 - [Alert][yarn_resourcemanager_webui] HA nameservice value is present but there are no aliases for {{yarn-site/yarn.resourcemanager.ha.rm-ids}}
INFO 2016-05-23 13:43:29,393 logger.py:67 - Host contains mounts: ['/', '/proc', '/sys', '/dev/pts', '/dev/shm', '/boot', '/home', '/proc/sys/fs/binfmt_misc'].
INFO 2016-05-23 13:43:29,397 logger.py:67 - Mount point for directory /hadoop/hdfs/data is /
INFO 2016-05-23 13:43:29,408 logger.py:67 - Execute['export HIVE_CONF_DIR='/usr/hdp/current/hive-metastore/conf/conf.server' ; hive --hiveconf hive.metastore.uris=thrift://HOSTNAME:9083                 --hiveconf hive.metastore.client.connect.retry.delay=1                 --hiveconf hive.metastore.failure.retries=1                 --hiveconf hive.metastore.connect.retries=1                 --hiveconf hive.metastore.client.socket.timeout=14                 --hiveconf hive.execution.engine=mr -e 'show databases;''] {'path': ['/bin/', '/usr/bin/', '/usr/sbin/', '/usr/hdp/current/hive-metastore/bin'], 'user': 'ambari-qa', 'timeout': 30}
INFO 2016-05-23 13:43:29,425 logger.py:67 - Execute['! beeline -u 'jdbc:hive2://HOSTNAME:10000/;transportMode=binary' -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL''] {'path': ['/bin/', '/usr/bin/', '/usr/lib/hive/bin/', '/usr/sbin/'], 'user': 'ambari-qa', 'timeout': 30}
INFO 2016-05-23 13:43:32,826 Heartbeat.py:78 - Building Heartbeat: {responseId = 32, timestamp = 1464000212826, commandsInProgress = False, componentsMapped = True}
INFO 2016-05-23 13:43:32,831 Controller.py:260 - Heartbeat response received (id = 33)
INFO 2016-05-23 13:43:32,831 RecoveryManager.py:217 - HIVE_SERVER needs recovery.
INFO 2016-05-23 13:43:42,831 Heartbeat.py:78 - Building Heartbeat: {responseId = 33, timestamp = 1464000222831, commandsInProgress = False, componentsMapped = True}
INFO 2016-05-23 13:43:42,875 Controller.py:260 - Heartbeat response received (id = 34)
INFO 2016-05-23 13:43:42,875 RecoveryManager.py:217 - HIVE_SERVER needs recovery.
INFO 2016-05-23 13:43:52,876 Heartbeat.py:78 - Building Heartbeat: {responseId = 34, timestamp = 1464000232875, commandsInProgress = False, componentsMapped = True}
INFO 2016-05-23 13:43:52,944 Controller.py:260 - Heartbeat response received (id = 35)

HDP version 2.3.2.0-2950 ambari version 2.1.2

Re: can't start hiveserver2 with ambari - beeline error Error: Could not open client transport with JDBC Uri

neomerdien — Mon, 23 May 2016 18:11:45 GMT

maybe this will help, but when starting hiveserver2 manually (/usr/bin/hive --service hiveserver2), I see its running with default memory values and not using the value in hive-site.xml, so maybe its why ambari having issues as well and i get the warning that i dont have heapsize set (16/05/23 14:06:54 WARN conf.HiveConf: HiveConf of name hive.heapsize does not exist):

root      8622 30797  4 14:06 pts/0    00:00:09 /usr/java/default/bin/java -Xmx256m -Dhdp.version=2.3.2.0-2950 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.3.2.0-2950 -Dhadoop.log.dir=/var/log/hadoop/root -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.3.2.0-2950/hadoop -Dhadoop.id.str=root -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.3.2.0-2950/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.3.2.0-2950/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx256m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.3.2.0-2950/hive/lib/hive-service-1.2.1.2.3.2.0-2950.jar org.apache.hive.service.server.HiveServer2

Re: can't start hiveserver2 with ambari - beeline error Error: Could not open client transport with JDBC Uri

sshimpi — Mon, 23 May 2016 19:52:40 GMT

@Andrey Nikitin

re-checking the above comment from @Kuldeep Kulkarni I see that you started hiveservice2 using cli and hence its not ambari managed or you are still able to see hiveserver2 stop in ambari.

Can you stop/kill the process of hiveserver2 from cli and try to start hive service from ambari ?

use this -

$ps -aef |grep hiveserver2 <- get pid from this command

$kill -9 <pid-of-hiveserver2-process>

restart hive service from ambari.

Re: can't start hiveserver2 with ambari - beeline error Error: Could not open client transport with JDBC Uri

neomerdien — Mon, 23 May 2016 20:01:35 GMT

@Sagar Shimpi i killed the process that was started manually and when i try to start hiveserver2 from ambari it does not start at all , thats why I opened this issue in the first place.

I have the following error in ambari:

Connection failed on host HOSTNAME:10000 
(Execution of '! beeline -u 'jdbc:hive2://HOSTNAME:10000/;transportMode=binary' -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL''
returned 1. Error: Could not open client transport with JDBC Uri: jdbc:hive2://

HOSTNAME:10000/;

transportMode=binary: java.net.ConnectException: Connection refused (state=08S01,code=0) Error: Could not open client transport with JDBC Uri: jdbc:hive2://

HOSTNAME:10000/;

transportMode=binary: java.net.ConnectException: Connection refused (state=08S01,code=0))

Re: can't start hiveserver2 with ambari - beeline error Error: Could not open client transport with JDBC Uri

deepesh1 — Mon, 23 May 2016 23:58:47 GMT

@Andrey Nikitin

To debug this issue it's important to look at the HiveServer2 log. It goes into ${hive.log.dir}. You can search for this log location on the Ambari UI by searching for "log" under the Services > Hive > Configs, it should show up in the section "Advanced hive-env" and corresponds to field name "Hive Log Dir". Make sure that directory exists and writable by hive user. In this directory you will also find hivemetastore.log corresponding to Metastore service.

The manual starting of HiveServer2 that you did

/usr/bin/hive --service hiveserver2

was performed as root user, that is not going to work, the service needs to be started as "hive" user. When service is started manually, Ambari doesn't know about it and hence it will show as stopped. I think the way Ambari detects a running HiveServer2 service is by looking for a PID file that it put in the first place when it started HiveServer2. So, starting the service manually confuses Ambari and it shows it as not running.

Typically when things are fine, running the following on hive host

ps -ef | grep "^hive"

should show two processes, one for Hive Metastore and other for HiveServer2.

Re: can't start hiveserver2 with ambari - beeline error Error: Could not open client transport with JDBC Uri

neomerdien — Tue, 24 May 2016 03:22:11 GMT

Hi @Deepesh

thanks for clarifying on how to start HiveServer2 manually.

the problem is that I am checking for HiveServer2 logs (in /var/log/hive) but the problem is that the logs there are empty. for example the metastore logs are there and if I restart it the logs updates with information successfully.

I know for a fact that its the right log location because when I restart HiveServer2 through ambari, the timestamp on those logs updates and corresponds to the time I try to start the service in ambari but the logs still show up as empty and don't provide any information.

hope that explains my situation.

Thanks.

Re: can't start hiveserver2 with ambari - beeline error Error: Could not open client transport with JDBC Uri

jyadav — Tue, 24 May 2016 22:10:30 GMT

Thanks @Andrey Nikitin for send us prompt replies.

This has been now resolved through https://community.hortonworks.com/questions/35195/how-can-i-check-ambari-start-services-commands.html#answer-35200 🙂

Re: can't start hiveserver2 with ambari - beeline error Error: Could not open client transport with JDBC Uri

muanang — Thu, 16 Jun 2016 09:01:47 GMT

I Had the same prob (with a different solution). Just sharing.

After I applied the smartsense recommendations I got an Alert from ambari (hive)

Fail: Execution of '! beeline -u 'jdbc:hive2://hivehost:10000/;transportMode=binary' -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'' returned 1. Error: Could not open client transport with JDBC Uri: jdbc:hive2://hivehost:10000/;transportMode=binary: java.net.ConnectException: Connection refused (state=08S01,code=0) Error: Could not open client transport with JDBC Uri: jdbc:hive2://hivehost:10000/;transportMode=binary: java.net.ConnectException: Connection refused (state=08S01,code=0) )

I've been reading all the posts and comments that are related to this prob. Thanks to you guys!

Upon checking on hiveserver2.log, the issue was related on queuename (I've setup a yarn scheduler). The fix was, to set a queuename for hive and add 'hive' to submit application to that queuename

Thanks again!

Re: can't start hiveserver2 with ambari - beeline error Error: Could not open client transport with JDBC Uri

erik_alfthan — Tue, 05 Dec 2017 07:50:28 GMT

Just adding another solution of the issue - I noticed that I could connect to localhost:3306 but not hostname.domain:3306 after deploying with "create new mysql"

Changing the bind address of mysql and restarting the mysql daemon resolved this issue

/# nano /etc/mysql/mysql.conf.d/mysqld.cnf

...
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address            = 0.0.0.0     # Changed from 127.0.0.1 to resolve hive connectivity issue

I dont know if this issue occurred to me because I used docker to deploy (getting some strange network side effects?) or if mysql changed default behaviour lately.

//Erik