Running spark programs as spark OS user

[#1]

Hi,

I am following Spark 1.6 tech preview tutorial as mentioned in the link http://hortonworks.com/hadoop-tutorial/apache-spark-1-6-technical-preview-with-hdp-2-3/

In one of the steps in mentions to switch to spark OS user by doing su spark.

Just wondering if there is any alternate way of running spark applications by proxy user. For example, if I am logging into the machine as user 'tom', who doesn't have sudo rights, then how to run spark applications through the shell without changing to spark user.

Many Thanks in advance

[#2]

you can run as tom, I don't see issues @Greenhorn Techie

[#3 · Accepted solution]

@Greenhorn Techie

You have to be careful before you switch users because you need to make sure that your id had proper permissions on HDFS to read and write. Answer is yes but again, make sure you do this

su - hdfs

hdfs dfs -mkdir /user/tom

hdfs dfs -chown -R tom:hdfs /user/tom

[#4]

@Neeraj Sabharwal

The reason I ask this question is because in the tech preview tutorial, its explicitly mentioned to change to 'spark' user before running the spark-shell commands. So wondering whats the need for it.

Assume, I have logged in as user 'tom' and have a corresponding folder on hdfs - /user/tom with read and write privileges. In that case, can I run spark-shell without changing to 'spark' user as mentioned in the link?

[#5]

@Greenhorn Techie answer is still yes

[#6]

Thanks @Artem Ervits for the clarification. I got confused because of the explicit mention of it in the tutorial.

[#7]

sure thing @Greenhorn Techie try it out and accept best answer. Also encourage you to publish an article on doing that.