Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

run sparksql on oozie but can't instantiate SessionHiveMetaStoreClient

run sparksql on oozie but can't instantiate SessionHiveMetaStoreClient

Explorer

 

Simple sparkSQL application works: (In CDH 5.8.0)

  1. fine with spark-submit
  2. run through oozie, generate below exception:
  3. What r we doing wrong?

 

Oozie exception:

 

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

 

 

Spark Code:----------------------------

import org.apache.spark._

import org.apache.spark.sql._

import org.apache.spark.sql.hive._

 

object SQLAction{

        def main(args:Array[String])

        {

        val conf = new SparkConf()

 

        val sc = new SparkContext(conf)

        val sqlc = new HiveContext(sc) // cause exception

        println("sparkSQL success") //not get run

 

        sqlc sql "select * from testcase_pos" collect

 

        println("select success")

        sc.stop

        }

}

----------------------------

 

What we tried:

 

  1. Add hive-site.xml as Job.xml. Hive-site.xml is stored under workspace. Same Exception
  2. Add conf.set("hive.metastore.uris","thrift://host:9083") into spark code. Not working
  3. We netstat the metastore thrift port but nothing is there
  4. …Could some one show me what's underline of oozie sparksql action?
1 REPLY 1

Re: run sparksql on oozie but can't instantiate SessionHiveMetaStoreClient

New Contributor

Hi maybe try uploading your hive-site.xml to hdfs and referancing it in the oozie options the gearwheel on the cell.

in the Options box  something like this  

--files ${nameNode}/Analytics/Spark_jars/hive-site.xml

 

Thanks