Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to fix Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hive/service/rpc/thrift/TCLIService$Iface

Highlighted

How to fix Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hive/service/rpc/thrift/TCLIService$Iface

New Contributor

I am trying to connect to Hive using plain scala code as below:

def getHiveConnection(): Connection = {
    val driver   = "org.apache.hive.jdbc.HiveDriver"
    val connection: Connection = null
    val conf = new Configuration()
    conf.set("hadoop.security.authentication", "Kerberos")
    System.setProperty("java.security.krb5.kdc", "ip-address")
    System.setProperty("java.security.krb5.realm", "DEV.COM")
    UserGroupInformation.loginUserFromKeytab("username@DEV.COM", "/home/username/username.keytab")
    try {
        Class.forName(driver)
        if((connection == null)  || connection.isClosed()) {
            val connection: Connection = DriverManager.getConnection(url, username, password)
        }
    } catch {
        case cnf:ClassNotFoundException => println("Invalid driver used. Check the settings.")
            cnf.printStackTrace()
        case e:Exception => e.printStackTrace()
    }
    connection
}

def exchange(): Unit = {
    val hiveCon   = getHiveConnection()
    val statement = hiveCon.createStatement()
    val drop      = statement.executeUpdate(s"ALTER TABLE schema.${k} drop partition(pname='${v}')")
    if (drop == 0) {
        val exchange  = statement.executeUpdate(s"ALTER TABLE schema.${k} EXCHANGE PARTITION (pname = '${v}') WITH TABLE stg.${v}_dummytable")
    }
}


Below are the contents from my build.sbt file:


libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "2.0.0" % "provided",
  "org.apache.spark" %% "spark-sql" % "2.0.0" % "provided",
  "org.json4s" %% "json4s-jackson" % "3.2.11" % "provided",
  "org.apache.httpcomponents" % "httpclient" % "4.5.3"
)

libraryDependencies += "org.postgresql" % "postgresql" % "42.1.4"
libraryDependencies += "org.apache.hadoop" % "hadoop-auth" % "2.6.0"
libraryDependencies += "org.apache.hadoop" % "hadoop-common" % "2.6.2"
libraryDependencies += "org.apache.hadoop" % "hadoop-core" % "1.2.1"
libraryDependencies += "org.apache.hadoop" % "hadoop-mapreduce-client-common" % "2.6.5"
libraryDependencies += "org.apache.hadoop" % "hadoop-mapreduce-client-core" % "2.6.5"
libraryDependencies += "org.apache.hadoop" % "hadoop-mapreduce-client-jobclient" % "2.6.5" % "provided"
libraryDependencies += "org.apache.hive" % "hive-jdbc" % "2.3.5"
libraryDependencies += "org.slf4j" % "slf4j-api" % "1.7.26"
libraryDependencies += "commons-cli" % "commons-cli" % "1.4"


The spark-submit command:

SPARK_MAJOR_VERSION=2 spark-submit --class com.partition.source.Loader --master=yarn --driver-class-path /home/username/jars/greenplum.jar:/home/username/jars/postgresql-42.1.4.jar:/home/username/test/inputdir/myjars/hive-jdbc-2.3.5.jar --conf spark.jars=/home/username/jars/greenplum.jar,/home/username/jars/postgresql-42.1.4.jar,/home/username/test/inputdir/myjars/hive-jdbc-2.3.5.jar --executor-cores 4 --executor-memory 4G --keytab /home/username/username.keytab --principal username@DEV.COM --files /usr/hdp/current/spark2-client/conf/hive-site.xml,connection.properties --name LOADER --conf spark.executor.extraClassPath=/home/username/jars/greenplum.jar <myjarname.jar>


The reason I am using scala code inside a spark program is that some of the operations such as "EXCHANGE PARTITION" are not supported in spark.sql. Hence I am trying to connect to Hive beeline and run the EXCHANGE command and get things done.

But when I submit the code, I get the driver not found exception as below:


Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hive/service/rpc/thrift/TCLIService$Iface
    at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
    at java.sql.DriverManager.getConnection(DriverManager.java:664)
    at java.sql.DriverManager.getConnection(DriverManager.java:247)
    at com.data.stages.ExchangePartition.getHiveConnection(ExchangePartition.scala:26)
    at com.data.stages.ExchangePartition.exchange(ExchangePartition.scala:40)
    at com.partition.source.Pickup$.main(Pickup.scala:123)
    at com.partition.source.Pickup.main(Pickup.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:782)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hive.service.rpc.thrift.TCLIService$Iface
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 16 more


Could anyone let me know if I am missing any dependency or is there any version mismatch ?