Created 09-05-2018 01:06 PM
Hello,
I have a cluster running on OpenStack. I need to load some data into DataFrame from HDFS:
val df_source = spark.read.parquet("hdfs://eureambarimaster1.local.eurecat.org:8020/user/hdfs/mydata/df_source.parquet")
I was always able to read data in this cluster without any problem. spark-shell
:
spark-shell Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 18/09/04 16:23:54 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041. 18/09/04 16:23:54 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042.Spark context Web UI available at http://192.168.0.9:4042 Spark context available as 'sc' (master = local[*], app id = local-1536078235081). Spark session available as 'spark'. Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_161) Type in expressions to have them evaluated. Type :help for more information. scala> val df_source = spark.read.parquet("hdfs://eureambarimaster1.local.eurecat.org:8020/user/hdfs/mydata/df_source.parquet") 18/09/04 16:26:13 WARN metastore: Failed to connect to the MetaStore Server... 18/09/04 16:28:22 WARN metastore: Failed to connect to the MetaStore Server... 18/09/04 16:30:30 WARN metastore: Failed to connect to the MetaStore Server... 18/09/04 16:30:31 WARN Hive: Failed to access metastore. This class should not accessed in runtime. org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1236) at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:174) at org.apache.hadoop.hive.ql.metadata.Hive.<clinit>(Hive.java:166) ...I read about this issue here: https://issues.apache.org/jira/browse/SPARK-15899
and tried the proposed solution, i.e. `spark-shell spark-shell --conf spark.sql.warehouse.dir=file:///home/centos/tmp`, but still the same result.