can't start hiveserver2 with ambari - beeline error Error: Could not open client transport with JDBC Uri

[#1]

Hi,

when i try to start hiveserver2 using ambari i get the following error:

Connection failed on host HOSTNAME:10000 (Execution of '! beeline -u 'jdbc:hive2://HOSTNAME:10000/;transportMode=binary' -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'' returned 1.

Error: Could not open client transport with JDBC Uri: jdbc:hive2://HOSTNAME:10000/; transportMode=binary: java.net.ConnectException: Connection refused (state=08S01,code=0)

Error: Could not open client transport with JDBC Uri: jdbc:hive2://HOSTNAME:10000/; transportMode=binary: java.net.ConnectException: Connection refused (state=08S01,code=0))

i cant connect locally using: beeline -u 'jdbc:hive2://

even though the service is down.

hive metastore and WebHCat server are running.

I understand that the reason i cant connect is because hiveserver2 is down, but interestingly enough hive-server2.log (/var/log/hive) remains blank when i start the service so I cant understand what it fails to start in the first place.

HDP version 2.3.2.0-2950

Ambari version 2.1.2

any help will be appreciated.

[#2]

@Andrey Nikitin

To debug this issue it's important to look at the HiveServer2 log. It goes into ${hive.log.dir}. You can search for this log location on the Ambari UI by searching for "log" under the Services > Hive > Configs, it should show up in the section "Advanced hive-env" and corresponds to field name "Hive Log Dir". Make sure that directory exists and writable by hive user. In this directory you will also find hivemetastore.log corresponding to Metastore service.

The manual starting of HiveServer2 that you did

/usr/bin/hive --service hiveserver2

was performed as root user, that is not going to work, the service needs to be started as "hive" user. When service is started manually, Ambari doesn't know about it and hence it will show as stopped. I think the way Ambari detects a running HiveServer2 service is by looking for a PID file that it put in the first place when it started HiveServer2. So, starting the service manually confuses Ambari and it shows it as not running.

Typically when things are fine, running the following on hive host

ps -ef | grep "^hive"

should show two processes, one for Hive Metastore and other for HiveServer2.

[#3]

Hi @Deepesh

thanks for clarifying on how to start HiveServer2 manually.

the problem is that I am checking for HiveServer2 logs (in /var/log/hive) but the problem is that the logs there are empty. for example the metastore logs are there and if I restart it the logs updates with information successfully.

I know for a fact that its the right log location because when I restart HiveServer2 through ambari, the timestamp on those logs updates and corresponds to the time I try to start the service in ambari but the logs still show up as empty and don't provide any information.

hope that explains my situation.

Thanks.

[#4 · Accepted solution]

Thanks @Andrey Nikitin for send us prompt replies.

This has been now resolved through https://community.hortonworks.com/questions/35195/how-can-i-check-ambari-start-services-commands.htm... 🙂

[#5]

I Had the same prob (with a different solution). Just sharing.

After I applied the smartsense recommendations I got an Alert from ambari (hive)

Fail: Execution of '! beeline -u 'jdbc:hive2://hivehost:10000/;transportMode=binary' -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'' returned 1. Error: Could not open client transport with JDBC Uri: jdbc:hive2://hivehost:10000/;transportMode=binary: java.net.ConnectException: Connection refused (state=08S01,code=0) Error: Could not open client transport with JDBC Uri: jdbc:hive2://hivehost:10000/;transportMode=binary: java.net.ConnectException: Connection refused (state=08S01,code=0) )

I've been reading all the posts and comments that are related to this prob. Thanks to you guys!

Upon checking on hiveserver2.log, the issue was related on queuename (I've setup a yarn scheduler). The fix was, to set a queuename for hive and add 'hive' to submit application to that queuename

Thanks again!

[#6]

Hi

Just adding another solution of the issue - I noticed that I could connect to localhost:3306 but not hostname.domain:3306 after deploying with "create new mysql"

Changing the bind address of mysql and restarting the mysql daemon resolved this issue

/# nano /etc/mysql/mysql.conf.d/mysqld.cnf

...
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address            = 0.0.0.0     # Changed from 127.0.0.1 to resolve hive connectivity issue

I dont know if this issue occurred to me because I used docker to deploy (getting some strange network side effects?) or if mysql changed default behaviour lately.

//Erik