Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Spark job failed when new HiveContext object

avatar
Contributor

We are using HDP 2.3.4. I also followed the instructions below.

  • spark-submit \
  • --class <Your.class.name> \
  • --master yarn-cluster \
  • --num-executors 1 \
  • --driver-memory 1g \
  • --executor-memory 1g \
  • --executor-cores 1 \
  • --files /usr/hdp/current/spark-client/conf/hive-site.xml \
  • --jars /usr/hdp/current/spark-client/lib/datanucleus-api-jdo-3.2.6.jar,/usr/hdp/current/spark-client/lib/datanucleus-rdbms-3.2.9.jar,/usr/hdp/current/spark-client/lib/datanucleus-core-3.2.10.jar \
  • target/YOUR_JAR-1.0.0-SNAPSHOT.jar "show tables""select * from your_table"
  • Here is the callstack:

    16/09/06 15:20:35 WARN Hive: Failed to access metastore. This class should not accessed in runtime. org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1236) at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:174) at org.apache.hadoop.hive.ql.metadata.Hive.<clinit>(Hive.java:166) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503) at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:193) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.spark.sql.hive.client.IsolatedClientLoader.liftedTree1$1(IsolatedClientLoader.scala:183) at org.apache.spark.sql.hive.client.IsolatedClientLoader.<init>(IsolatedClientLoader.scala:179) at org.apache.spark.sql.hive.HiveContext.metadataHive$lzycompute(HiveContext.scala:228) at org.apache.spark.sql.hive.HiveContext.metadataHive(HiveContext.scala:187) at org.apache.spark.sql.hive.HiveContext.setConf(HiveContext.scala:394) at org.apache.spark.sql.hive.HiveContext.defaultOverrides(HiveContext.scala:176) at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:179) at com.cbt.ingest.tsz.TSZIngestApp$delayedInit$body.apply(TSZIngestApp.scala:50) at scala.Function0$class.apply$mcV$sp(Function0.scala:40) at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12) at scala.App$anonfun$main$1.apply(App.scala:71) at scala.App$anonfun$main$1.apply(App.scala:71) at scala.collection.immutable.List.foreach(List.scala:318) at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)

    1 ACCEPTED SOLUTION

    avatar
    Contributor

    The problem is resolved by using SQLContext in spark application code. Thanks for quick response.

    View solution in original post

    16 REPLIES 16

    avatar

    Could you post your spark code?

    avatar

    Also, could you post any additional context in the stack trace. Are there additional exceptions?

    avatar
    Contributor

    It failed at the beginning of my code to new a HiveContext object...

    log.warn("Running Master: " + master.toString()) val sparkConf = new SparkConf().setAppName(APP_NAME) .setMaster(master) val sc = SparkContext.getOrCreate(sparkConf) val sqlContext = new SQLContext(sc) val hiveSqlContext = new org.apache.spark.sql.hive.HiveContext(sc)

    avatar
    Contributor

    Actually I noticed there is successfully connect to metastore at the beginning and then later on it tried to connect it again and then failed.

    16/09/07 13:14:11 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 1297829 for jzhou5 on ha-hdfs:hd0

    16/09/07 13:14:12 INFO metastore: Trying to connect to metastore with URI thrift://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

    16/09/07 13:14:12 INFO metastore: Connected to metastore.

    16/09/07 13:14:12 INFO Client: Uploading resource file:/usr/hdp/2.3.4.0-3485/spark/lib/spark-assembly-1.5.2.2.3.4.0-3485-hadoop2.7.1.2.3.4.0-3485.jar -> hdfs://hd0/user/.sparkStaging/application_1473231848025_1554/spark-assembly-1.5.2.2.3.4.0-3485-hadoop2.7.1.2.3.4.0-3485.jar

    ....

    16/09/07 13:14:24 INFO HiveContext: Initializing execution hive, version 1.2.1

    16/09/07 13:14:24 INFO ClientWrapper: Inspected Hadoop version: 2.7.1.2.3.4.0-3485

    16/09/07 13:14:24 INFO ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.7.1.2.3.4.0-3485

    16/09/07 13:14:24 INFO metastore: Trying to connect to metastore with URI thrift://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

    16/09/07 13:14:24 INFO metastore: Connected to metastore.

    16/09/07 13:14:24 INFO SessionState: Created local directory: /tmp/2a51b1f7-5c87-4b2a-95c6-bc7eb06d900b_resources

    16/09/07 13:14:24 INFO SessionState: Created HDFS directory: /tmp/hive/jzhou5/2a51b1f7-5c87-4b2a-95c6-

    ....

    16/09/07 13:14:25 INFO metastore: Trying to connect to metastore with URI thrift://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

    16/09/07 13:14:25 WARN Hive: Failed to access metastore. This class should not accessed in runtime.

    org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

    at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1236)

    at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:174)

    at org.apache.hadoop.hive.ql.metadata.Hive.<clinit>(Hive.java:166)

    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)

    at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:193)

    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

    avatar
    Contributor

    Here are the full error output of the job:

    16/09/07 14:21:36 WARN Hive: Failed to access metastore. This class should not accessed in runtime. org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1236) at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:174) at org.apache.hadoop.hive.ql.metadata.Hive.<clinit>(Hive.java:166) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503) at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:193) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.spark.sql.hive.client.IsolatedClientLoader.liftedTree1$1(IsolatedClientLoader.scala:183) at org.apache.spark.sql.hive.client.IsolatedClientLoader.<init>(IsolatedClientLoader.scala:179) at org.apache.spark.sql.hive.HiveContext.metadataHive$lzycompute(HiveContext.scala:228) at org.apache.spark.sql.hive.HiveContext.metadataHive(HiveContext.scala:187) at org.apache.spark.sql.hive.HiveContext.setConf(HiveContext.scala:394) at org.apache.spark.sql.hive.HiveContext.defaultOverrides(HiveContext.scala:176) at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:179) at com.cbt.ingest.tsz.TSZIngestApp$delayedInit$body.apply(TSZIngestApp.scala:53) at scala.Function0$class.apply$mcV$sp(Function0.scala:40) at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12) at scala.App$anonfun$main$1.apply(App.scala:71) at scala.App$anonfun$main$1.apply(App.scala:71) at scala.collection.immutable.List.foreach(List.scala:318) at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32) at scala.App$class.main(App.scala:71) at com.cbt.ingest.tsz.TSZIngestApp.main(TSZIngestApp.scala:29) at com.cbt.ingest.tsz.GenericTSZIngest.main(TSZIngestApp.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:685) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024) at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1234) ... 34 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521) ... 40 more Caused by: java.lang.IllegalStateException: Error finding hadoop SASL properties at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge23.getHadoopSaslProperties(HadoopThriftAuthBridge23.java:103) at org.apache.hadoop.hive.metastore.MetaStoreUtils.getMetaStoreSaslProperties(MetaStoreUtils.java:1588) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:401) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:236) at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74) ... 45 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge23.getHadoopSaslProperties(HadoopThriftAuthBridge23.java:98) ... 49 more Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.util.StringUtils.toUpperCase(Ljava/lang/String;)Ljava/lang/String; at org.apache.hadoop.security.SaslPropertiesResolver.setConf(SaslPropertiesResolver.java:69) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.security.SaslPropertiesResolver.getInstance(SaslPropertiesResolver.java:58) ... 54 more 16/09/07 14:21:36 INFO metastore: Trying to connect to metastore with URI thrift://xxxxxxxxxxxxxxxxxxx:9083 Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522) at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:193) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.spark.sql.hive.client.IsolatedClientLoader.liftedTree1$1(IsolatedClientLoader.scala:183) at org.apache.spark.sql.hive.client.IsolatedClientLoader.<init>(IsolatedClientLoader.scala:179) at org.apache.spark.sql.hive.HiveContext.metadataHive$lzycompute(HiveContext.scala:228) at org.apache.spark.sql.hive.HiveContext.metadataHive(HiveContext.scala:187) at org.apache.spark.sql.hive.HiveContext.setConf(HiveContext.scala:394) at org.apache.spark.sql.hive.HiveContext.defaultOverrides(HiveContext.scala:176) at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:179) at com.cbt.ingest.tsz.TSZIngestApp$delayedInit$body.apply(TSZIngestApp.scala:53) at scala.Function0$class.apply$mcV$sp(Function0.scala:40) at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12) at scala.App$anonfun$main$1.apply(App.scala:71) at scala.App$anonfun$main$1.apply(App.scala:71) at scala.collection.immutable.List.foreach(List.scala:318) at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32) at scala.App$class.main(App.scala:71) at com.cbt.ingest.tsz.TSZIngestApp.main(TSZIngestApp.scala:29) at com.cbt.ingest.tsz.GenericTSZIngest.main(TSZIngestApp.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:685) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503) ... 31 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521) ... 37 more Caused by: java.lang.IllegalStateException: Error finding hadoop SASL properties at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge23.getHadoopSaslProperties(HadoopThriftAuthBridge23.java:103) at org.apache.hadoop.hive.metastore.MetaStoreUtils.getMetaStoreSaslProperties(MetaStoreUtils.java:1588) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:401) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:236) at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74) ... 42 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge23.getHadoopSaslProperties(HadoopThriftAuthBridge23.java:98) ... 46 more Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.util.StringUtils.toUpperCase(Ljava/lang/String;)Ljava/lang/String; at org.apache.hadoop.security.SaslPropertiesResolver.setConf(SaslPropertiesResolver.java:69) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.security.SaslPropertiesResolver.getInstance(SaslPropertiesResolver.java:58) ... 51 more 16/09/07 14:21:36 INFO SparkContext: Invoking stop() from shutdown hook

    avatar
    Super Collaborator

    Is Kerberos enabled on the cluster?

    avatar
    Contributor

    The problem is resolved by using SQLContext in spark application code. Thanks for quick response.

    avatar
    Rising Star

    Hi @Jay Zhou

    Can you be a bit more specific what you have changed? What did you exactly do with this line?

    val hiveSqlContext = new org.apache.spark.sql.hive.HiveContext(sc)

    I have a similar problem where I get an error

    WARN Hive: Failed to access metastore. This class should not accessed in runtime.

    but this is only when I run the job via Oozie. When I use spark submit the code works so I guess the dependencies are right.

    Do you have any idea what can cause this?

    avatar
    Contributor

    Actually the problem still exists since I have to use HiveContext. I just noticed ClientWrapper inspected different hadoop versions, first one is correct and the second one is wrong.. highlighted below. Could this be the root cause?

    16/09/07 15:47:54 INFO BlockManagerMasterEndpoint: Registering block manager xxxxxxxx

    16/09/07 15:47:54 INFO HiveContext: Initializing execution hive, version 1.2.1

    16/09/07 15:47:54 INFO ClientWrapper: Inspected Hadoop version: 2.7.1.2.3.4.0-3485

    16/09/07 15:47:54 INFO ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.7.1.2.3.4.0-3485

    16/09/07 15:47:54 INFO metastore: Trying to connect to metastore with URI thrift://xxxxxxxxxxxxxxxxxxxxxxxx

    16/09/07 15:47:54 INFO metastore: Connected to metastore.

    16/09/07 15:47:54 INFO SessionState: Created local directory: /tmp/1d43c90d-da99-4970-80fd-31c9ad9a8d4d_resources

    16/09/07 15:47:54 INFO SessionState: Created HDFS directory: /tmp/hive/jzhou5/1d43c90d-da99-4970-80fd-31c9ad9a8d4d

    16/09/07 15:47:54 INFO SessionState: Created local directory: /tmp/jzhou5/1d43c90d-da99-4970-80fd-31c9ad9a8d4d

    16/09/07 15:47:54 INFO SessionState: Created HDFS directory: /tmp/hive/jzhou5/1d43c90d-da99-4970-80fd-31c9ad9a8d4d/_tmp_space.db

    16/09/07 15:47:54 INFO HiveContext: default warehouse location is /user/hive/warehouse

    16/09/07 15:47:54 INFO HiveContext: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.

    16/09/07 15:47:54 INFO ClientWrapper: Inspected Hadoop version: 2.2.0

    16/09/07 15:47:54 INFO ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.2.0

    16/09/07 15:47:55 INFO deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces

    16/09/07 15:47:55 INFO deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize