HIVE connection refused critical alert

[#1]

I have only one alert (critical) in the system , coming from HIVE and only thing that's not working is I cant connect to hive2 database using beeline. I can connect to hive user using MySQL and I can see all the tables under hive db .

the error I am seeing in Ambari is follows:

(err_msg) Fail: Execution of '! beeline -u
'jdbc:hive2://hadoop2.tolls.dot.state.fl.us:10000/;transportMode=binary' -e ''
2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL''
returned 1. Error: Could not open client transport with JDBC Uri:
jdbc:hive2://hadoop2.tolls.dot.state.fl.us:10000/;transportMode=binary:
java.net.ConnectException: Connection refused (state=08S01,code=0) Error: Could
not open client transport with JDBC Uri:
jdbc:hive2://hadoop2.tolls.dot.state.fl.us:10000/;transportMode=binary:
java.net.ConnectException: Connection refused (state=08S01,code=0) ) 

[#2]

are you able to reach this hadoop2.tolls.dot.state.fl.us and port 10000 from the node where you are running beeline?

[#3]

service is not running ? which service is using this 10000 port?

I read the solution of the similar issue suggesting the following but I don't know how to do it

Kill your hiveserver2 & metastore & Restart them again. ????
check hiveserver2 & metastore are running using 'jps' command. ????

[root@hadoop1 admin]# telnet hadoop2 10000
Trying 10.100.44.16...
telnet: connect to address 10.100.44.16: Connection refused
[root@hadoop1 admin]#

here is the output of jps command on hadoop2 where HIVE is installed.

7829 NodeManager
6194 gateway.jar
5679 Main
7292 Main
6014 Kafka
6530 HistoryServer
14238 Jps
2618 AmbariServer
6922 Main
4290 QuorumPeerMain
5868 HMaster
8753 Main
4178 DataNode
7167 Main
4462 NameNode
7478 HRegionServer
[root@hadoop1 admin]# jps
7829 NodeManager
6194 gateway.jar
5679 Main
7292 Main
6014 Kafka
6530 HistoryServer
2618 AmbariServer
6922 Main
4290 QuorumPeerMain
5868 HMaster
14398 Jps
8753 Main
4178 DataNode
7167 Main
4462 NameNode
7478 HRegionServer

[#4]

I checked and both hiveserver2 and metastore processes are running on the server hadoop2,

I still can't connect via beeline , and HDC is showing a critical alert on HIVE.

[root@hadoop2 admin]# ps -ef | grep metastore
hive      6178     1  1 14:21 ?        00:00:08 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.111.x86_64/bin/java -Xmx1024m -Dhdp.version=2.4.3.0-227 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.4.3.0-227 -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.4.3.0-227/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.4.3.0-227/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -XX:MaxPermSize=512m -Xmx1489m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.4.3.0-227/hive/lib/hive-service-1.2.1000.2.4.3.0-227.jar org.apache.hadoop.hive.metastore.HiveMetaStore -hiveconf hive.log.file=hivemetastore.log -hiveconf hive.log.dir=/var/log/hive
hive      8525     1  1 14:22 ?        00:00:10 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.111.x86_64/bin/java -Xmx1024m -Dhdp.version=2.4.3.0-227 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.4.3.0-227 -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.4.3.0-227/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.4.3.0-227/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -XX:MaxPermSize=512m -Xmx4467m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.4.3.0-227/hive/lib/hive-service-1.2.1000.2.4.3.0-227.jar org.apache.hive.service.server.HiveServer2 --hiveconf hive.aux.jars.path=file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar -hiveconf hive.metastore.uris=  -hiveconf hive.log.file=hiveserver2.log -hiveconf hive.log.dir=/var/log/hive
root     12000 11237  0 14:32 pts/0    00:00:00 grep metastore
[root@hadoop2 admin]#
[root@hadoop2 admin]#
[root@hadoop2 admin]#
[root@hadoop2 admin]# ps -ef | grep hiveserver2
hive      8525     1  1 14:22 ?        00:00:10 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.111.x86_64/bin/java -Xmx1024m -Dhdp.version=2.4.3.0-227 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.4.3.0-227 -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.4.3.0-227/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.4.3.0-227/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -XX:MaxPermSize=512m -Xmx4467m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.4.3.0-227/hive/lib/hive-service-1.2.1000.2.4.3.0-227.jar org.apache.hive.service.server.HiveServer2 --hiveconf hive.aux.jars.path=file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar -hiveconf hive.metastore.uris=  -hiveconf hive.log.file=hiveserver2.log -hiveconf hive.log.dir=/var/log/hive
root     12009 11237  0 14:32 pts/0    00:00:00 grep hiveserver2
[root@hadoop2 admin]#
[root@hadoop2 admin]#

[#5]

@Sami Ahmad

can you post

lsof -i:10000 # it should be return the process id of you hiveserver2

telnet hadoop2.tolls.dot.state.fl.us 10000

netstat -talnp| grep <hive server2 pid>

[#6]

@Sami Ahmad

Can paste out for this

ps -ef |grep hive

[#7]

Did you also looked inside /var/log/hive/hive-server2.out file after starting HS2 from Ambari?

[#8]

[root@hadoop2 admin]# ps -ef |grep hive hive 6178 1 0 14:21 ? 00:00:08 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.111.x86_64/bin/java -Xmx1024m -Dhdp.version=2.4.3.0-227 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.4.3.0-227 -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.4.3.0-227/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.4.3.0-227/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -XX:MaxPermSize=512m -Xmx1489m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.4.3.0-227/hive/lib/hive-service-1.2.1000.2.4.3.0-227.jar org.apache.hadoop.hive.metastore.HiveMetaStore -hiveconf hive.log.file=hivemetastore.log -hiveconf hive.log.dir=/var/log/hive hive 8525 1 1 14:22 ? 00:00:11 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.111.x86_64/bin/java -Xmx1024m -Dhdp.version=2.4.3.0-227 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.4.3.0-227 -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.4.3.0-227/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.4.3.0-227/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -XX:MaxPermSize=512m -Xmx4467m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.4.3.0-227/hive/lib/hive-service-1.2.1000.2.4.3.0-227.jar org.apache.hive.service.server.HiveServer2 --hiveconf hive.aux.jars.path=file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar -hiveconf hive.metastore.uris= -hiveconf hive.log.file=hiveserver2.log -hiveconf hive.log.dir=/var/log/hive hcat 8973 1 0 14:22 ? 00:00:06 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.111.x86_64/bin/java -Xmx1024m -Dhdp.version=2.4.3.0-227 -Djava.net.preferIPv4Stack=true -Dwebhcat.log.dir=/var/log/webhcat/ -Dlog4j.configuration=file:///usr/hdp/2.4.3.0-227/hive-hcatalog/sbin/../etc/webhcat/webhcat-log4j.properties -Dhdp.version=2.4.3.0-227 -Dhadoop.log.dir=/var/log/hadoop/hcat -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.4.3.0-227/hadoop -Dhadoop.id.str=hcat -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.4.3.0-227/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -XX:MaxPermSize=512m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.4.3.0-227/hive-hcatalog/sbin/../share/webhcat/svr/lib/hive-webhcat-1.2.1000.2.4.3.0-227.jar org.apache.hive.hcatalog.templeton.Main

[#9]

hive.zipthere are many files in this folder . I am uploading them all

[#10]

The hive user didn't have write access to /tmp/hive .set required permissions and have a try. . I can see this in hiveserver2 log