Hello,
I have a cluster running on OpenStack. I need to load some data into DataFrame from HDFS:
val df_source = spark.read.parquet("hdfs://eureambarimaster1.local.eurecat.org:8020/user/hdfs/mydata/df_source.parquet")I was always able to read data in this cluster without any problem. However ,now I've started getting the following error message from spark-shell:
spark-shell
Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
18/09/04 16:23:54 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
18/09/04 16:23:54 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042.Spark context Web UI available at http://192.168.0.9:4042
Spark context available as 'sc' (master = local[*], app id = local-1536078235081).
Spark session available as 'spark'.
Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_161)
Type in expressions to have them evaluated.
Type :help for more information.
scala> val df_source = spark.read.parquet("hdfs://eureambarimaster1.local.eurecat.org:8020/user/hdfs/mydata/df_source.parquet")
18/09/04 16:26:13 WARN metastore: Failed to connect to the MetaStore Server...
18/09/04 16:28:22 WARN metastore: Failed to connect to the MetaStore Server...
18/09/04 16:30:30 WARN metastore: Failed to connect to the MetaStore Server...
18/09/04 16:30:31 WARN Hive: Failed to access metastore.
This class should not accessed in runtime. org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1236) at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:174) at org.apache.hadoop.hive.ql.metadata.Hive.<clinit>(Hive.java:166) ...I read about this issue here: https://issues.apache.org/jira/browse/SPARK-15899
and tried the proposed solution, i.e. `spark-shell spark-shell --conf spark.sql.warehouse.dir=file:///home/centos/tmp`, but still the same result.
