Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Read data into DataFrame gives error in Spark Shell: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

Highlighted

Read data into DataFrame gives error in Spark Shell: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

Contributor

Hello,

I have a cluster running on OpenStack. I need to load some data into DataFrame from HDFS:

val df_source = spark.read.parquet("hdfs://eureambarimaster1.local.eurecat.org:8020/user/hdfs/mydata/df_source.parquet")

I was always able to read data in this cluster without any problem. However ,now I've started getting the following error message from spark-shell:

spark-shell	
Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel).	For SparkR, use setLogLevel(newLevel).	
18/09/04 16:23:54 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
18/09/04 16:23:54 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042.Spark context Web UI available at http://192.168.0.9:4042	
Spark context available as 'sc' (master = local[*], app id = local-1536078235081).	
Spark session available as 'spark'.	
Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_161)	
Type in expressions to have them evaluated.
Type :help for more information. 
scala>	val df_source = spark.read.parquet("hdfs://eureambarimaster1.local.eurecat.org:8020/user/hdfs/mydata/df_source.parquet")	
18/09/04 16:26:13 WARN metastore: Failed to connect to the MetaStore Server...
18/09/04 16:28:22 WARN metastore: Failed to connect to the MetaStore Server...	
18/09/04 16:30:30 WARN metastore: Failed to connect to the MetaStore Server...	
18/09/04 16:30:31 WARN Hive: Failed to access metastore.	
This class should not accessed in runtime. org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1236) at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:174) at org.apache.hadoop.hive.ql.metadata.Hive.<clinit>(Hive.java:166) ...

I read about this issue here: https://issues.apache.org/jira/browse/SPARK-15899

and tried the proposed solution, i.e. `spark-shell spark-shell --conf spark.sql.warehouse.dir=file:///home/centos/tmp`, but still the same result.

Don't have an account?
Coming from Hortonworks? Activate your account here