can't start hiveserver2 with ambari - beeline error Error: Could not open client transport with JDBC Uri

[#1]

Hi,

when i try to start hiveserver2 using ambari i get the following error:

Connection failed on host HOSTNAME:10000 (Execution of '! beeline -u 'jdbc:hive2://HOSTNAME:10000/;transportMode=binary' -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'' returned 1.

Error: Could not open client transport with JDBC Uri: jdbc:hive2://HOSTNAME:10000/; transportMode=binary: java.net.ConnectException: Connection refused (state=08S01,code=0)

Error: Could not open client transport with JDBC Uri: jdbc:hive2://HOSTNAME:10000/; transportMode=binary: java.net.ConnectException: Connection refused (state=08S01,code=0))

i cant connect locally using: beeline -u 'jdbc:hive2://

even though the service is down.

hive metastore and WebHCat server are running.

I understand that the reason i cant connect is because hiveserver2 is down, but interestingly enough hive-server2.log (/var/log/hive) remains blank when i start the service so I cant understand what it fails to start in the first place.

HDP version 2.3.2.0-2950

Ambari version 2.1.2

any help will be appreciated.

[#2]

@Andrey Nikitin

Enable hive debug and check if you are able to see logs.

Also once hive is up try with below commands -

# beeline

>!connect jdbc:hive2://HOSTNAME:10000/default

Else try -

>!connect jdbc:hive2://<host>:<port>/<db>;transportMode=http;httpPath=<http_endpoint>

where:-

<http_endpoint> is the corresponding HTTP endpoint configured in hive-site.xml. Default value is cliservice.

Default port for HTTP transport mode is 10001.

If you are having kerberized cluster then user -

>!connect jdbc:hive2://<host>:10000/;principal=<Server_Principal_of_HiveServer2>

[#3]

Hi @Sagar Shimpi

how do i enable Hive debug?

Thanks.

[#4]

@Andrey Nikitin

In addition to what @Sagar Shimpi said, can you please run below command on hiveserver2 host to see DEBUG output on console? it should give you a hint

Please run below command as hive user:

/usr/bin/hive --service hiveserver2 --hiveconf hive.root.logger=DEBUG,console 

[#5]

Hi @Sagar Shimpi , @Kuldeep Kulkarni

some updates:

I've started hiveserver2 like you mentioned: /usr/bin/hive --service hiveserver2 --hiveconf hive.root.logger=DEBUG,console and it started without any issues and there was no apparent errors in the log, only these two warnings but i dont think its relevant: WARNING: Use "yarn jar" to launch YARN applications. 16/05/23 10:41:17 WARN conf.HiveConf: HiveConf of name hive.heapsize does not exist

after the service was started via the above command, I tried to "start" the service in ambari and it connected succesfully, also via the CLI with the following command: beeline -u 'jdbc:hive2://HOSTNAME:10000/;transportMode=binary'

when I tried to stop the service from ambari, it showed me that the services was stopped but when I checked the server it was still running

all of the above leads me to the conclusion that there is some problem between ambari and hiveserver2, its like its unable to start/stop the hiveserver2 service, and that is probably the reason why the log files are not update

please advise on what could an issue between ambari and running the services? i dont have problems with other services running from ambari on the same server.

Thanks.

[#6]

@Andrey Nikitin

Just give a try of restarting ambari server and agent(on which hiveserver2 runs)

Also pls let us know the version of ambari and hdp you are using.

[#7]

@Sagar Shimpi

i restarted the ambari server and agent on the hive machine and the outcome is the same.

this is the log from ambari-agent:

INFO 2016-05-23 13:42:52,555 RecoveryManager.py:217 - HIVE_SERVER needs recovery.
INFO 2016-05-23 13:42:54,316 RecoveryManager.py:217 - HIVE_SERVER needs recovery.
INFO 2016-05-23 13:43:02,556 Heartbeat.py:78 - Building Heartbeat: {responseId = 29, timestamp = 1464000182555, commandsInProgress = False, componentsMapped = True}
INFO 2016-05-23 13:43:02,577 Controller.py:260 - Heartbeat response received (id = 30)
INFO 2016-05-23 13:43:02,577 RecoveryManager.py:217 - HIVE_SERVER needs recovery.
INFO 2016-05-23 13:43:12,578 Heartbeat.py:78 - Building Heartbeat: {responseId = 30, timestamp = 1464000192578, commandsInProgress = False, componentsMapped = True}
INFO 2016-05-23 13:43:12,779 Controller.py:260 - Heartbeat response received (id = 31)
INFO 2016-05-23 13:43:12,780 RecoveryManager.py:217 - HIVE_SERVER needs recovery.
INFO 2016-05-23 13:43:22,781 Heartbeat.py:78 - Building Heartbeat: {responseId = 31, timestamp = 1464000202781, commandsInProgress = False, componentsMapped = True}
INFO 2016-05-23 13:43:22,825 Controller.py:260 - Heartbeat response received (id = 32)
INFO 2016-05-23 13:43:22,826 RecoveryManager.py:217 - HIVE_SERVER needs recovery.
WARNING 2016-05-23 13:43:29,367 base_alert.py:417 - [Alert][yarn_resourcemanager_webui] HA nameservice value is present but there are no aliases for {{yarn-site/yarn.resourcemanager.ha.rm-ids}}
INFO 2016-05-23 13:43:29,393 logger.py:67 - Host contains mounts: ['/', '/proc', '/sys', '/dev/pts', '/dev/shm', '/boot', '/home', '/proc/sys/fs/binfmt_misc'].
INFO 2016-05-23 13:43:29,397 logger.py:67 - Mount point for directory /hadoop/hdfs/data is /
INFO 2016-05-23 13:43:29,408 logger.py:67 - Execute['export HIVE_CONF_DIR='/usr/hdp/current/hive-metastore/conf/conf.server' ; hive --hiveconf hive.metastore.uris=thrift://HOSTNAME:9083                 --hiveconf hive.metastore.client.connect.retry.delay=1                 --hiveconf hive.metastore.failure.retries=1                 --hiveconf hive.metastore.connect.retries=1                 --hiveconf hive.metastore.client.socket.timeout=14                 --hiveconf hive.execution.engine=mr -e 'show databases;''] {'path': ['/bin/', '/usr/bin/', '/usr/sbin/', '/usr/hdp/current/hive-metastore/bin'], 'user': 'ambari-qa', 'timeout': 30}
INFO 2016-05-23 13:43:29,425 logger.py:67 - Execute['! beeline -u 'jdbc:hive2://HOSTNAME:10000/;transportMode=binary' -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL''] {'path': ['/bin/', '/usr/bin/', '/usr/lib/hive/bin/', '/usr/sbin/'], 'user': 'ambari-qa', 'timeout': 30}
INFO 2016-05-23 13:43:32,826 Heartbeat.py:78 - Building Heartbeat: {responseId = 32, timestamp = 1464000212826, commandsInProgress = False, componentsMapped = True}
INFO 2016-05-23 13:43:32,831 Controller.py:260 - Heartbeat response received (id = 33)
INFO 2016-05-23 13:43:32,831 RecoveryManager.py:217 - HIVE_SERVER needs recovery.
INFO 2016-05-23 13:43:42,831 Heartbeat.py:78 - Building Heartbeat: {responseId = 33, timestamp = 1464000222831, commandsInProgress = False, componentsMapped = True}
INFO 2016-05-23 13:43:42,875 Controller.py:260 - Heartbeat response received (id = 34)
INFO 2016-05-23 13:43:42,875 RecoveryManager.py:217 - HIVE_SERVER needs recovery.
INFO 2016-05-23 13:43:52,876 Heartbeat.py:78 - Building Heartbeat: {responseId = 34, timestamp = 1464000232875, commandsInProgress = False, componentsMapped = True}
INFO 2016-05-23 13:43:52,944 Controller.py:260 - Heartbeat response received (id = 35)

HDP version 2.3.2.0-2950 ambari version 2.1.2

[#8]

maybe this will help, but when starting hiveserver2 manually (/usr/bin/hive --service hiveserver2), I see its running with default memory values and not using the value in hive-site.xml, so maybe its why ambari having issues as well and i get the warning that i dont have heapsize set (16/05/23 14:06:54 WARN conf.HiveConf: HiveConf of name hive.heapsize does not exist):

root      8622 30797  4 14:06 pts/0    00:00:09 /usr/java/default/bin/java -Xmx256m -Dhdp.version=2.3.2.0-2950 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.3.2.0-2950 -Dhadoop.log.dir=/var/log/hadoop/root -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.3.2.0-2950/hadoop -Dhadoop.id.str=root -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.3.2.0-2950/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.3.2.0-2950/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx256m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.3.2.0-2950/hive/lib/hive-service-1.2.1.2.3.2.0-2950.jar org.apache.hive.service.server.HiveServer2

[#9]

@Andrey Nikitin

re-checking the above comment from @Kuldeep Kulkarni I see that you started hiveservice2 using cli and hence its not ambari managed or you are still able to see hiveserver2 stop in ambari.

Can you stop/kill the process of hiveserver2 from cli and try to start hive service from ambari ?

use this -

$ps -aef |grep hiveserver2 <- get pid from this command

$kill -9 <pid-of-hiveserver2-process>

restart hive service from ambari.

[#10]

@Sagar Shimpi i killed the process that was started manually and when i try to start hiveserver2 from ambari it does not start at all , thats why I opened this issue in the first place.

I have the following error in ambari:

Connection failed on host HOSTNAME:10000 
(Execution of '! beeline -u 'jdbc:hive2://HOSTNAME:10000/;transportMode=binary' -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL''
returned 1. Error: Could not open client transport with JDBC Uri: jdbc:hive2://

HOSTNAME:10000/;

transportMode=binary: java.net.ConnectException: Connection refused (state=08S01,code=0) Error: Could not open client transport with JDBC Uri: jdbc:hive2://

HOSTNAME:10000/;

transportMode=binary: java.net.ConnectException: Connection refused (state=08S01,code=0))