Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Unable to connect to Hive server using JDBC connection

Unable to connect to Hive server using JDBC connection

Contributor

I am trying to connect to Hive server from scala code as below.

def getHiveConnection(): Connection = {
println("Building Hive connection..")
val driver = "org.apache.hive.jdbc.HiveDriver"
val user = "user"
val pwd = "pwd
val url = "jdbc:hive2://ip-00-000-000-000.ec2.internal:00000/dbname;principal=hive/ip-00-000-000-000.ec2.internal@DEV.COM"
var connection: Connection = null

val conf = new Configuration()
conf.set("hadoop.security.authentication", "Kerberos")
UserGroupInformation.setConfiguration(conf)

try {
println("Setting the driver..")
Class.forName(driver)
println("pre connection")
if((connection == null) || connection.isClosed()) {
connection = DriverManager.getConnection(url, user, pwd)
println("Hive connection eshtablished.")
}
} catch {
case cnf:ClassNotFoundException => println("Invalid driver used. Check the settings.")
cnf.printStackTrace()
case e:Exception => println("Other exception.")
e.printStackTrace()
}
connection
}

I create a jar file from the program on IntelliJ and then run the jar using spar-submit as I need to run some sql that is not supported by SPARK.

spark-submit:

SPARK_MAJOR_VERSION=2 spark-submit --class com.package.program.Begin --master=yarn --conf spark.ui.port=4090 --driver-class-path /home/username/testlib/inputdir/myjars/hive-jdbc-2.3.5.jar --conf spark.jars=/home/username/testlib/inputdir/myjars/hive-jdbc-2.3.5.jar --executor-cores 4 --executor-memory 4G --keytab /home/username/username.keytab --principal username@DEV.COM --files /$SPARK_HOME/conf/hive-site.xml,connection.properties --name Splinter splinter_2.11-0.1.jar

 

When I submit the code, it fails with the exception:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hive/service/rpc/thrift/TCLIService$Iface

 

To be precise, the exception comes at the line:

connection = DriverManager.getConnection(url, user, pwd)

Dependencies I added in the SBT files can be seen below:

name := "Splinter"
version := "0.1"
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.0.0",
"org.apache.spark" %% "spark-sql" % "2.0.0",
"org.json4s" %% "json4s-jackson" % "3.2.11",
"org.apache.httpcomponents" % "httpclient" % "4.5.3",
"org.apache.spark" %% "spark-hive" % "2.0.0",
)
libraryDependencies += "org.postgresql" % "postgresql" % "42.1.4"
libraryDependencies += "org.apache.hadoop" % "hadoop-auth" % "2.6.0"
libraryDependencies += "org.apache.hadoop" % "hadoop-common" % "2.6.2"
libraryDependencies += "org.apache.hadoop" % "hadoop-core" % "1.2.1"
libraryDependencies += "org.apache.hadoop" % "hadoop-mapreduce-client-common" % "2.6.5"
libraryDependencies += "org.apache.hadoop" % "hadoop-mapreduce-client-core" % "2.6.5"
libraryDependencies += "org.apache.hadoop" % "hadoop-mapreduce-client-jobclient" % "2.6.5" % "provided"
libraryDependencies += "org.apache.hive" % "hive-jdbc" % "2.3.5"
libraryDependencies += "org.apache.hive" % "hive-common" % "2.3.5"
libraryDependencies += "org.apache.hive" % "hive-metastore" % "2.3.5"
libraryDependencies += "org.apache.hive" % "hive-service" % "2.3.5"
libraryDependencies += "org.slf4j" % "slf4j-api" % "1.7.26"
libraryDependencies += "commons-cli" % "commons-cli" % "1.4"
libraryDependencies += "org.apache.hive" % "hive-service-rpc" % "2.1.0"
libraryDependencies += "org.apache.hive" % "hive-cli" % "2.3.5"
libraryDependencies += "org.apache.hive" % "hive-exec" % "2.3.4" excludeAll
ExclusionRule(organization = "org.pentaho")

 

Along with the dependencies, I moved all the jars from the dir via `--jars` in spark-submit and that didn't work either.

The full exception stack could be seen below:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hive/service/rpc/thrift/TCLIService$Iface
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
at com.data.stages.ExchangePartition.getHiveConnection(ExchangePartition.scala:30)
at com.data.stages.ExchangePartition.exchange(ExchangePartition.scala:44)
at com.partition.source.Pickup$.main(Pickup.scala:124)
at com.partition.source.Pickup.main(Pickup.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:782)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hive.service.rpc.thrift.TCLIService$Iface
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 16 more

 

Could anyone let me know what dependencies am I missing in the sbt file ?
If not, what could be the mistake I am doing here since, the same type of code works in Jave with the same libraries(dependencies) in the project and I couldn't understand what is wrong here ?
Any help is much appreciated.

Don't have an account?
Coming from Hortonworks? Activate your account here