Thank you!   I have change it.   Also I have changed my paths. Because the path is for directory and not for a file. I have also added a / to my path. Now I get results which I have expected. I changed "setMaster to "local" because it is just a small Cloudera VM without cluster.   This is a simple Spark script which can be executed in hue per Spark editor:   from pyspark import SparkContext, SparkConf appNameTEST ="my first working application" conf = SparkConf().setAppName(appNameTEST).setMaster("local") sc = SparkContext(conf=conf) text_file = sc.textFile("hdfs:///user/hive/warehouse/TEST/FilePath") counts = text_file.flatMap(lambda line: line.split(" ")) \              .map(lambda word: (word, 1)) \              .reduceByKey(lambda a, b: a + b) counts.saveAsTextFile("hdfs:///user/hive/warehouse/TEST/RESULT") ... View more

hdfs fsck will give you answers for your questions, there are multiple command line parameters, you can show the block's location as well. ... View more

@bgooley @Tomas79   I ended up loging in a user when the EMR is launching in the bootstrap action.I did this via curl commands.This will avoid any user being given super user status.   For anyone needing guidance on the workaround you could follow the below steps.   1) Curl command to get the cookie.txt(it has the session id and csrf token) file. 2) Curl command to login(You have to grep from the cookie.txt file the session id and csrftoken)    If anyone has a better idea please let me know ... View more

It worth to check if the use case is actually suited for using HDFS's NFS Gateway role[1] which is designed for such remote cluster access. [1] - Adding and Configuring an NFS Gateway - https://www.cloudera.com/documentation/enterprise/5-12-x/topics/admin_hdfs_nfsgateway.html ... View more