hive connection via zookeeper

[#1]

Hi, 

I'm trying to connect to hive from zookeeper, I'm getting a java null pointer exception with beeline cmd. currently using CDH 5.16.2 version. 

I have a valid kerberos ticket:

beeline --verbose=true -u "jdbc:hive2://ZK1:2181,ZK2:2181,ZK3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveServer2;transportMode=binary;httpPath=cliservice;"
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
issuing: !connect jdbc:hive2://ZK1:2181,ZK2:2181,ZK3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveServer2;transportMode=binary;httpPath=cliservice; '' [passwd stripped]
scan complete in 2ms
Connecting to jdbc:hive2://ZK1:2181,ZK2:2181,ZK3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveServer2;transportMode=binary;httpPath=cliservice;
java.lang.NullPointerException
at org.apache.thrift.transport.TSocket.open(TSocket.java:209)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:203)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:168)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:146)
at org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:211)
at org.apache.hive.beeline.Commands.connect(Commands.java:1529)
at org.apache.hive.beeline.Commands.connect(Commands.java:1424)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:52)
at org.apache.hive.beeline.BeeLine.execCommandWithPrefix(BeeLine.java:1139)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1178)
at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:818)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:898)
at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:518)
at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:226)
at org.apache.hadoop.util.RunJar.main(RunJar.java:141)
Beeline version 1.1.0-cdh5.16.2 by Apache Hive
0: jdbc:hive2://ZK1:2181, (closed)>

[#2]

Try and use it after this modification. I succeeded like this.

[You]

jdbc:hive2://zk=hadoopcluster01:2181,hadoopcluster02:2181/hiveserver2

[My Proposal]

jdbc:hive2://zk=hadoopcluster01:2181/hiveserver2,hadoopcluster02:2181/hiveserver2

[#3]

Thanks Lewis, but it didn't worked!!

Caused by: java.lang.IllegalArgumentException: A HostProvider may not be empty!
at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:72)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445)
at org.apache.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.java:29)
at org.apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.java:154)
at org.apache.curator.HandleHolder$1.getZooKeeper(HandleHolder.java:94)
at org.apache.curator.HandleHolder.internalClose(HandleHolder.java:128)
at org.apache.curator.HandleHolder.closeAndClear(HandleHolder.java:71)
at org.apache.curator.ConnectionState.close(ConnectionState.java:114)
... 30 more
Error: org.apache.hive.jdbc.ZooKeeperHiveClientException: Unable to read HiveServer2 configs from ZooKeeper (state=,code=0)
Beeline version 1.1.0-cdh5.16.2 by Apache Hive
beeline>

[#4]

@raghu9raghavend 

I see that you are using the below connect string:

jdbc:hive2://ZK1:2181,ZK2:2181,ZK3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveServer2;transportMode=binary;httpPath=cliservice;

You are using both "transportMode=binary" and then you are providing the "httpPath=cliservice" as well.

"httpPath=cliservice" should be provided only when "transportMode=http".

Thus, if your Hiveserver2 is running in HTTP mode, please set the "transportMode=http" in the connect string.

jdbc:hive2://ZK1:2181,ZK2:2181,ZK3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveServer2;transportMode=http;httpPath=cliservice;

Alternatively, if you are have HiveServer2 in binary mode, remove both the options.

jdbc:hive2://ZK1:2181,ZK2:2181,ZK3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveServer2;

[#5]

Thanks Rohit, I tried both options but it did not helped me. 

Apparently cloudera engineer is saying that this option(i.e, connecting to hive via zookeeper) will not be supported in CDH 5.16.2 version. Is it true?

beeline -u "jdbc:hive2://ZK1:2181,ZK2:2181,ZK3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hive_zookeeper_namespace_hive;transportMode=http;httpPath=cliservice;"
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
scan complete in 1ms
Connecting to jdbc:hive2://ZK1:2181,ZK2:2181,ZK3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hive_zookeeper_namespace_hive;transportMode=http;httpPath=cliservice;
20/01/17 09:27:26 [main]: INFO jdbc.HiveConnection: Failed to connect to null:0
20/01/17 09:27:27 [main]: INFO jdbc.HiveConnection: Failed to connect to null:0
20/01/17 09:27:27 [main]: INFO jdbc.HiveConnection: Failed to connect to null:0
20/01/17 09:27:27 [main]: INFO jdbc.HiveConnection: Failed to connect to null:0
20/01/17 09:27:27 [main]: INFO jdbc.HiveConnection: Failed to connect to null:0
20/01/17 09:27:27 [main]: INFO jdbc.HiveConnection: Failed to connect to null:0
20/01/17 09:27:27 [main]: INFO jdbc.HiveConnection: Failed to connect to null:0
20/01/17 09:27:27 [main]: INFO jdbc.HiveConnection: Failed to connect to null:0
20/01/17 09:27:27 [main]: INFO jdbc.HiveConnection: Failed to connect to null:0
20/01/17 09:27:27 [main]: INFO jdbc.HiveConnection: Failed to connect to null:0
20/01/17 09:27:27 [main]: INFO jdbc.HiveConnection: Failed to connect to null:0
20/01/17 09:27:27 [main]: INFO jdbc.HiveConnection: Failed to connect to null:0
20/01/17 09:27:27 [main]: INFO jdbc.HiveConnection: Failed to connect to null:0
20/01/17 09:27:27 [main]: INFO jdbc.HiveConnection: Failed to connect to null:0
20/01/17 09:27:27 [main]: INFO jdbc.HiveConnection: Failed to connect to null:0
Error: Could not open client transport for any of the Server URI's in ZooKeeper: Unable to read HiveServer2 configs from ZooKeeper (state=08S01,code=0)
Beeline version 1.1.0-cdh5.16.2 by Apache Hive
beeline>

beeline -u "jdbc:hive2://ZK1:2181,ZK2:2181,ZK3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hive_zookeeper_namespace_hive;"
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
scan complete in 1ms
Connecting to jdbc:hive2://ZK1:2181,ZK2:2181,ZK3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hive_zookeeper_namespace_hive;
java.lang.NullPointerException
Beeline version 1.1.0-cdh5.16.2 by Apache Hive
0: jdbc:hive2://ZK1:2181, (closed)>

Thanks,

Raghu.

[#6]

Hi, do you know this error reason and how to fix it?I have the same error

[#7]

Hi Raghu,

I think we have missed the service discovery 

Please check with the below command and confirm

Host names and port numbers  where your zookeeper is running . We also call it as Zookeeper ensemble.
We can easily get this value from property hive.zookeeper.quorum in Hive.

zookeeper's default port number is 2181. You can get zookeeper host names from Zookeeper configuration files also.


Specify service discovery mode using serviceDiscoveryMode=zooKeeper .


Specify zookeeper namespace as hivesever2. This is the value of hive.server2.zookeeper.namespace property in Hive.


We are using below connection string .

jdbc:hive2://datanode1:2181,master1:2181,master2:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2

[#8]

Thanks!

[#9]

@raghu9raghavend  I think I caught the issue, you are using binary and proving httppath

jdbc:hive2://ZK1:2181,ZK2:2181,ZK3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveServer2;transportMode=binary;httpPath=cliservice; 

Please check,if below set to http or binary 

hive.server2.transport.mode

Change the url:

http:

jdbc:hive2://ZK1:2181,ZK2:2181,ZK3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;transportMode=http;httpPath=cliservice; 

binary:

jdbc:hive2://ZK1:2181,ZK2:2181,ZK3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2

Change 

 hiveSever2 ==> hiveserver2

[#10]

Thanks for the responses. This is a old one and resolved long back.