Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: The Cloudera Community will undergo maintenance on Saturday, August 17 at 12:00am PDT. See more info here.

How to fix Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hive/service/rpc/th

Highlighted

How to fix Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hive/service/rpc/th

Contributor

I am trying to connect to Hive using plain scala code as below:

    def getHiveConnection(): Connection = {
        val driver   = "org.apache.hive.jdbc.HiveDriver"
        val connection: Connection = null
        val conf = new Configuration()
        conf.set("hadoop.security.authentication", "Kerberos")
        System.setProperty("java.security.krb5.kdc", "ip-address")
        System.setProperty("java.security.krb5.realm", "DEV.COM")
        UserGroupInformation.loginUserFromKeytab("username@DEV.COM", "/home/username/username.keytab")
        try {
            Class.forName(driver)
            if((connection == null)  || connection.isClosed()) {
                val connection: Connection = DriverManager.getConnection(url, username, password)
            }
        } catch {
            case cnf:ClassNotFoundException => println("Invalid driver used. Check the settings.")
                cnf.printStackTrace()
            case e:Exception => e.printStackTrace()
        }
        connection
    }
    def exchange(): Unit = {
        val hiveCon   = getHiveConnection()
        val statement = hiveCon.createStatement()
        val drop      = statement.executeUpdate(s"ALTER TABLE schema.${k} drop partition(pname='${v}')")
        if (drop == 0) {
            val exchange  = statement.executeUpdate(s"ALTER TABLE schema.${k} EXCHANGE PARTITION (pname = '${v}') WITH TABLE stg.${v}_dummytable")
        }
    }

Below are the contents from my build.sbt file:

 

    libraryDependencies ++= Seq(
      "org.apache.spark" %% "spark-core" % "2.0.0" % "provided",
      "org.apache.spark" %% "spark-sql" % "2.0.0" % "provided",
      "org.json4s" %% "json4s-jackson" % "3.2.11" % "provided",
      "org.apache.httpcomponents" % "httpclient" % "4.5.3"
    )
    libraryDependencies += "org.postgresql" % "postgresql" % "42.1.4"
    libraryDependencies += "org.apache.hadoop" % "hadoop-auth" % "2.6.0"
    libraryDependencies += "org.apache.hadoop" % "hadoop-common" % "2.6.2"
    libraryDependencies += "org.apache.hadoop" % "hadoop-core" % "1.2.1"
    libraryDependencies += "org.apache.hadoop" % "hadoop-mapreduce-client-common" % "2.6.5"
    libraryDependencies += "org.apache.hadoop" % "hadoop-mapreduce-client-core" % "2.6.5"
    libraryDependencies += "org.apache.hadoop" % "hadoop-mapreduce-client-jobclient" % "2.6.5" % "provided"
    libraryDependencies += "org.apache.hive" % "hive-jdbc" % "2.3.5"
    libraryDependencies += "org.slf4j" % "slf4j-api" % "1.7.26"
    libraryDependencies += "commons-cli" % "commons-cli" % "1.4"


The spark-submit command:

 

SPARK_MAJOR_VERSION=2 spark-submit --class com.partition.source.Loader --master=yarn --driver-class-path /home/username/jars/greenplum.jar:/home/username/jars/postgresql-42.1.4.jar:/home/username/test/inputdir/myjars/hive-jdbc-2.3.5.jar --conf spark.jars=/home/username/jars/greenplum.jar,/home/username/jars/postgresql-42.1.4.jar,/home/username/test/inputdir/myjars/hive-jdbc-2.3.5.jar --executor-cores 4 --executor-memory 4G --keytab /home/username/username.keytab --principal username@DEV.COM --files /usr/hdp/current/spark2-client/conf/hive-site.xml,connection.properties --name LOADER --conf spark.executor.extraClassPath=/home/username/jars/greenplum.jar <myjarname.jar>

The reason I am using scala code inside a spark program is that some of the operations such as "EXCHANGE PARTITION" are not supported in spark.sql. Hence I am trying to connect to Hive beeline and run the EXCHANGE command and get things done.

 

But when I submit the code, I get the driver not found exception as below:

 

    Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hive/service/rpc/thrift/TCLIService$Iface
        at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:247)
        at com.data.stages.ExchangePartition.getHiveConnection(ExchangePartition.scala:26)
        at com.data.stages.ExchangePartition.exchange(ExchangePartition.scala:40)
        at com.partition.source.Pickup$.main(Pickup.scala:123)
        at com.partition.source.Pickup.main(Pickup.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:782)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
    Caused by: java.lang.ClassNotFoundException: org.apache.hive.service.rpc.thrift.TCLIService$Iface
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 16 more

Could anyone let me know if I am missing any dependency or is there any version mismatch ?