Hello, I am having some issues when trying to fetch data from hive using spark 2. In my classpath I have the hive.site.xml, core-site.xml and hdfs-site.xml. If I try to access the parquet file directly from HDFS using the following code, it works: val spark = SparkSession .builder() .master("local") .appName("myname") .enableHiveSupport() .getOrCreate() //avoids linkage exception RuntimeDelegate.setInstance(new RuntimeDelegateImpl()) spark.read.parquet("/PATH/partition_name=20180525/*").show(10) The result: +----------+--------------------+----------+---------+---------+-----------+ | col1| col2| col3| col4| col5| col6| +----------+--------------------+----------+---------+---------+-----------+ |2018-05-24|ABC1234 | ABC2| ABC123|bbbbbbbbb| 0| +----------+--------------------+----------+---------+---------+-----------+ But if I try to use spark sql to query HIVE: spark.table("mydatabase.mytable").limit(1).show() I have the following error: Failed to connect to /184.6.241.44:1019 for block, add to deadNodes and continue. java.net.ConnectException: Connection timed out: no further information I can query hive through Knox or using JDBC, so everything looks fine, but this. What could cause this problem ? Any idea ? Thank you. ... View more

In my company we secure HDFS with Knox for external services, so when I am developing locally I need to go through Knox to fetch HDFS files. In our projects we are using Spark 1.6 and we had to implement a custom FileSystem to wrap this Knox access. As I am developing a new project using Spark 2.1, I was wondering if there is an easier way to fetch this HDFS data without implementing a custom file system. What's the right way to do this? ... View more