question Re: HIVE connection refused critical alert in Archives of Support Questions (Read Only)

HIVE connection refused critical alert

aliyesami — Wed, 14 Sep 2016 00:36:32 GMT

I have only one alert (critical) in the system , coming from HIVE and only thing that's not working is I cant connect to hive2 database using beeline. I can connect to hive user using MySQL and I can see all the tables under hive db .

the error I am seeing in Ambari is follows:

(err_msg) Fail: Execution of '! beeline -u
'jdbc:hive2://hadoop2.tolls.dot.state.fl.us:10000/;transportMode=binary' -e ''
2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL''
returned 1. Error: Could not open client transport with JDBC Uri:
jdbc:hive2://hadoop2.tolls.dot.state.fl.us:10000/;transportMode=binary:
java.net.ConnectException: Connection refused (state=08S01,code=0) Error: Could
not open client transport with JDBC Uri:
jdbc:hive2://hadoop2.tolls.dot.state.fl.us:10000/;transportMode=binary:
java.net.ConnectException: Connection refused (state=08S01,code=0) )

Re: HIVE connection refused critical alert

rajkumar_singh — Wed, 14 Sep 2016 00:38:11 GMT

are you able to reach this hadoop2.tolls.dot.state.fl.us and port 10000 from the node where you are running beeline?

Re: HIVE connection refused critical alert

aliyesami — Wed, 14 Sep 2016 00:50:13 GMT

service is not running ? which service is using this 10000 port?

I read the solution of the similar issue suggesting the following but I don't know how to do it

Kill your hiveserver2 & metastore & Restart them again. ????
check hiveserver2 & metastore are running using 'jps' command. ????

[root@hadoop1 admin]# telnet hadoop2 10000
Trying 10.100.44.16...
telnet: connect to address 10.100.44.16: Connection refused
[root@hadoop1 admin]#

here is the output of jps command on hadoop2 where HIVE is installed.

7829 NodeManager
6194 gateway.jar
5679 Main
7292 Main
6014 Kafka
6530 HistoryServer
14238 Jps
2618 AmbariServer
6922 Main
4290 QuorumPeerMain
5868 HMaster
8753 Main
4178 DataNode
7167 Main
4462 NameNode
7478 HRegionServer
[root@hadoop1 admin]# jps
7829 NodeManager
6194 gateway.jar
5679 Main
7292 Main
6014 Kafka
6530 HistoryServer
2618 AmbariServer
6922 Main
4290 QuorumPeerMain
5868 HMaster
14398 Jps
8753 Main
4178 DataNode
7167 Main
4462 NameNode
7478 HRegionServer

Re: HIVE connection refused critical alert

chennuri_gouris — Wed, 14 Sep 2016 01:28:08 GMT

@Sami Ahmad

Can paste out for this

ps -ef |grep hive

Re: HIVE connection refused critical alert

chennuri_gouris — Wed, 14 Sep 2016 01:30:55 GMT

Did you also looked inside /var/log/hive/hive-server2.out file after starting HS2 from Ambari?

Re: HIVE connection refused critical alert

aliyesami — Wed, 14 Sep 2016 01:33:41 GMT

I checked and both hiveserver2 and metastore processes are running on the server hadoop2,

I still can't connect via beeline , and HDC is showing a critical alert on HIVE.

[root@hadoop2 admin]# ps -ef | grep metastore
hive      6178     1  1 14:21 ?        00:00:08 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.111.x86_64/bin/java -Xmx1024m -Dhdp.version=2.4.3.0-227 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.4.3.0-227 -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.4.3.0-227/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.4.3.0-227/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -XX:MaxPermSize=512m -Xmx1489m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.4.3.0-227/hive/lib/hive-service-1.2.1000.2.4.3.0-227.jar org.apache.hadoop.hive.metastore.HiveMetaStore -hiveconf hive.log.file=hivemetastore.log -hiveconf hive.log.dir=/var/log/hive
hive      8525     1  1 14:22 ?        00:00:10 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.111.x86_64/bin/java -Xmx1024m -Dhdp.version=2.4.3.0-227 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.4.3.0-227 -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.4.3.0-227/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.4.3.0-227/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -XX:MaxPermSize=512m -Xmx4467m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.4.3.0-227/hive/lib/hive-service-1.2.1000.2.4.3.0-227.jar org.apache.hive.service.server.HiveServer2 --hiveconf hive.aux.jars.path=file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar -hiveconf hive.metastore.uris=  -hiveconf hive.log.file=hiveserver2.log -hiveconf hive.log.dir=/var/log/hive
root     12000 11237  0 14:32 pts/0    00:00:00 grep metastore
[root@hadoop2 admin]#
[root@hadoop2 admin]#
[root@hadoop2 admin]#
[root@hadoop2 admin]# ps -ef | grep hiveserver2
hive      8525     1  1 14:22 ?        00:00:10 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.111.x86_64/bin/java -Xmx1024m -Dhdp.version=2.4.3.0-227 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.4.3.0-227 -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.4.3.0-227/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.4.3.0-227/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -XX:MaxPermSize=512m -Xmx4467m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.4.3.0-227/hive/lib/hive-service-1.2.1000.2.4.3.0-227.jar org.apache.hive.service.server.HiveServer2 --hiveconf hive.aux.jars.path=file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar -hiveconf hive.metastore.uris=  -hiveconf hive.log.file=hiveserver2.log -hiveconf hive.log.dir=/var/log/hive
root     12009 11237  0 14:32 pts/0    00:00:00 grep hiveserver2
[root@hadoop2 admin]#
[root@hadoop2 admin]#

Re: HIVE connection refused critical alert

aliyesami — Wed, 14 Sep 2016 01:38:11 GMT

[root@hadoop2 admin]# ps -ef |grep hive hive 6178 1 0 14:21 ? 00:00:08 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.111.x86_64/bin/java -Xmx1024m -Dhdp.version=2.4.3.0-227 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.4.3.0-227 -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.4.3.0-227/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.4.3.0-227/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -XX:MaxPermSize=512m -Xmx1489m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.4.3.0-227/hive/lib/hive-service-1.2.1000.2.4.3.0-227.jar org.apache.hadoop.hive.metastore.HiveMetaStore -hiveconf hive.log.file=hivemetastore.log -hiveconf hive.log.dir=/var/log/hive hive 8525 1 1 14:22 ? 00:00:11 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.111.x86_64/bin/java -Xmx1024m -Dhdp.version=2.4.3.0-227 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.4.3.0-227 -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.4.3.0-227/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.4.3.0-227/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -XX:MaxPermSize=512m -Xmx4467m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.4.3.0-227/hive/lib/hive-service-1.2.1000.2.4.3.0-227.jar org.apache.hive.service.server.HiveServer2 --hiveconf hive.aux.jars.path=file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar -hiveconf hive.metastore.uris= -hiveconf hive.log.file=hiveserver2.log -hiveconf hive.log.dir=/var/log/hive hcat 8973 1 0 14:22 ? 00:00:06 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.111.x86_64/bin/java -Xmx1024m -Dhdp.version=2.4.3.0-227 -Djava.net.preferIPv4Stack=true -Dwebhcat.log.dir=/var/log/webhcat/ -Dlog4j.configuration=file:///usr/hdp/2.4.3.0-227/hive-hcatalog/sbin/../etc/webhcat/webhcat-log4j.properties -Dhdp.version=2.4.3.0-227 -Dhadoop.log.dir=/var/log/hadoop/hcat -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.4.3.0-227/hadoop -Dhadoop.id.str=hcat -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.4.3.0-227/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -XX:MaxPermSize=512m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.4.3.0-227/hive-hcatalog/sbin/../share/webhcat/svr/lib/hive-webhcat-1.2.1000.2.4.3.0-227.jar org.apache.hive.hcatalog.templeton.Main

Re: HIVE connection refused critical alert

aliyesami — Wed, 14 Sep 2016 01:44:01 GMT

hive.zipthere are many files in this folder . I am uploading them all

Re: HIVE connection refused critical alert

chennuri_gouris — Wed, 14 Sep 2016 01:59:05 GMT

The hive user didn't have write access to /tmp/hive .set required permissions and have a try. . I can see this in hiveserver2 log

Re: HIVE connection refused critical alert

aliyesami — Wed, 14 Sep 2016 02:28:34 GMT

I added the permissions for hive user on /tmp/hive , still no luck ..

[hive@hadoop2 ~]$ id
uid=502(hive) gid=502(hadoop) groups=502(hadoop)
[hive@hadoop2 ~]$
[hive@hadoop2 ~]$ uname -a > a.a
[hive@hadoop2 ~]$ hdfs dfs -copyFromLocal a.a /tmp/hive/b.b
[hive@hadoop2 ~]$

Re: HIVE connection refused critical alert

aliyesami — Wed, 14 Sep 2016 02:29:46 GMT

I also tried this command , looks like something is not good in the database ?

[root@hadoop2 java]# metatool -listFSRoot
WARNING: Use "yarn jar" to launch YARN applications.
Initializing HiveMetaTool..
16/09/13 15:24:05 INFO metastore.ObjectStore: ObjectStore, initialize called
16/09/13 15:24:05 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored
16/09/13 15:24:05 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
16/09/13 15:24:05 ERROR Datastore.Schema: Failed initialising database.
Unable to open a test connection to the given database. JDBC url = jdbc:mysql://hadoop2.tolls.dot.state.fl.us/hive?createDatabaseIfNotExist=true, username = hive. Terminating connection pool (set lazyInit to true if you expect to start your database after your app). Original Exception: ------
java.sql.SQLException: Access denied for user 'hive'@'hadoop2.tolls.dot.state.fl.us' (using password: YES)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073)

Re: HIVE connection refused critical alert

chennuri_gouris — Wed, 14 Sep 2016 02:45:13 GMT

Can you run this, So that we can exactly know whats gi\oing on

/usr/bin/hive --service hiveserver2 --hiveconf hive.root.logger=DEBUG

Re: HIVE connection refused critical alert

aliyesami — Wed, 14 Sep 2016 02:46:02 GMT

hive.zipI bounced all the servers and restarted all components , attaching the new log files can you please see if you still see the hive user permission issues on /tmp/hive ? I am still getting permission denied.

Re: HIVE connection refused critical alert

aliyesami — Wed, 14 Sep 2016 03:42:04 GMT

hi gouri the hiveserver2 is already running on the hive node hadoop2. you want me to kill the process and run the one you gave in nohup mode so it runs from background?

[root@hadoop2 hive]# ps -ef | grep hiveserver2
hive      8474     1  2 16:33 ?        00:00:09 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.111.x86_64/bin/java -Xmx1024m -Dhdp.version=2.4.3.0-227 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.4.3.0-227 -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.4.3.0-227/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.4.3.0-227/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -XX:MaxPermSize=512m -Xmx4467m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.4.3.0-227/hive/lib/hive-service-1.2.1000.2.4.3.0-227.jar org.apache.hive.service.server.HiveServer2 --hiveconf hive.aux.jars.path=file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar -hiveconf hive.metastore.uris=  -hiveconf hive.log.file=hiveserver2.log -hiveconf hive.log.dir=/var/log/hive
root     10843  2790  0 16:40 pts/0    00:00:00 grep hiveserver2
[root@hadoop2 hive]#

Re: HIVE connection refused critical alert

chennuri_gouris — Wed, 14 Sep 2016 03:45:26 GMT

Please run below command as hive user: /usr/bin/hive --service hiveserver2 --hiveconf hive.root.logger=DEBUG . yes kill hiveserver2 process on hadoop2 and run this on the same node.

Re: HIVE connection refused critical alert

rajkumar_singh — Wed, 14 Sep 2016 05:57:15 GMT

@Sami Ahmad

can you post

lsof -i:10000 # it should be return the process id of you hiveserver2

telnet hadoop2.tolls.dot.state.fl.us 10000

netstat -talnp| grep <hive server2 pid>

Re: HIVE connection refused critical alert

aliyesami — Wed, 14 Sep 2016 10:35:08 GMT

Gouri you were right it was the privileges issue on Linux /tmp/hive folder , I was changing the permission of the hdfs /tmp/hive folder . I can access beeline now and can connect to the hive store , I have other issues though for which I will open a new post.

thanks for your help