Permissions issue Spark R

[#1]

In Rstudio I do:

library(sparklyr)
library(dplyr)
Sys.setenv(SPARK_HOME="/usr/hdp/current/spark2-client") # got from ambari spark2 configs
config <- spark_config()
sc <- spark_connect(master = "yarn-client", config = config, version = '2.1.0')

which gives:

 Failed during initialize_connection: org.apache.hadoop.security.AccessControlException: Permission denied: user=ed, access=WRITE, inode="/user/ed/.sparkStaging/application_1500959138473_0003":admin:hadoop:drwxr-xr-x

normally I fix this sort of problem with:

HADOOP_USER_NAME=hdfs hadoop fs -put 

but I do not know how to do this in R.

I thought maybe change ed's user and group to hdfs:

ed@master:~$ hdfs dfs -ls /user
Found 11 items
drwx------   - accumulo  hdfs            0 2017-05-14 15:38 /user/accumulo
drwxr-xr-x   - admin     hadoop          0 2017-06-27 06:52 /user/admin
drwxrwx---   - ambari-qa hdfs            0 2017-06-02 10:46 /user/ambari-qa
drwxr-xr-x   - admin     hadoop          0 2017-06-02 11:00 /user/ed
drwxr-xr-x   - hbase     hdfs            0 2017-05-14 15:35 /user/hbase
drwxr-xr-x   - hcat      hdfs            0 2017-05-14 15:44 /user/hcat
drwxr-xr-x   - hdfs      hdfs            0 2017-06-20 12:43 /user/hdfs
drwxr-xr-x   - hive      hdfs            0 2017-05-14 15:44 /user/hive
drwxrwxr-x   - oozie     hdfs            0 2017-05-14 15:46 /user/oozie
drwxrwxr-x   - spark     hdfs            0 2017-05-14 15:40 /user/spark
drwxr-xr-x   - zeppelin  hdfs            0 2017-07-24 09:29 /user/zeppelin

but I am worried as it is currently admin/hadoop and admin is how I log into Ambari. So I do not want to mess up other stuff.

Any help is much appreciated!

[#2 · Accepted solution]

@ed day

Hey you don't need to worry about the admin user stuff as I can see you have the "/user/admin" directory present in the HDFS with owner "admin".

Just go to HDFS, login to hdfs user and change the ownership of directory for 'ed' as,

# su hdfs
# hdfs dfs -chown ed:hdfs /user/ed

Let me know if this helps.

[#3]

Did the job, thanks!