Error connecting to Hive from Spark

[#1]

Hello everybody,

I am able to connect to beeline, using !connect, with following jdbc string

jdbc:hive2://<server_name>:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 no problem.

However I cannot connect using exactly same jdbc string from Spark

Code:

val currentSession = SparkSession .builder() .appName("mainHive") .master("local[*]") .getOrCreate();

val df = currentSession.read .format("jdbc") .option("url",<same_url>) .option("user", <user_name>) .option("password", <password>) .option("dbTable","base") .load();

Error:

18/08/10 16:16:05 INFO state.ConnectionStateManager: State change: CONNECTED 18/08/10 16:16:05 INFO zookeeper.ZooKeeper: Session: 0x364f5a35f66441d closed 18/08/10 16:16:05 INFO zookeeper.ClientCnxn: EventThread shut down Exception in thread "main" java.sql.SQLException: Could not open client transport for any of the Server URI's in ZooKeeper: Unable to read HiveServer2 uri from ZooKeeper at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:219) at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:176) at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)

Any idea would be appreciated

[#2]

Any reason why you want to establish a jdbc connection to hive? IIRC there were some issues while accessing hive via jdbc through spark.

Spark can directly connect to hive metastore. Just make sure you have hive-site.xml in /etc/spark/conf/* and create a hive context. Then you can directly access the hive tables. Ref : https://spark.apache.org/docs/1.6.1/sql-programming-guide.html#starting-point-sqlcontext