question Re: Running spark programs as spark OS user in Archives of Support Questions (Read Only)

Running spark programs as spark OS user

techiegreenhorn — Thu, 04 Feb 2016 23:26:33 GMT

Hi,

I am following Spark 1.6 tech preview tutorial as mentioned in the link http://hortonworks.com/hadoop-tutorial/apache-spark-1-6-technical-preview-with-hdp-2-3/

In one of the steps in mentions to switch to spark OS user by doing su spark.

Just wondering if there is any alternate way of running spark applications by proxy user. For example, if I am logging into the machine as user 'tom', who doesn't have sudo rights, then how to run spark applications through the shell without changing to spark user.

Many Thanks in advance

Re: Running spark programs as spark OS user

aervits — Thu, 04 Feb 2016 23:27:33 GMT

you can run as tom, I don't see issues @Greenhorn Techie

Re: Running spark programs as spark OS user

nsabharwal — Thu, 04 Feb 2016 23:29:42 GMT

@Greenhorn Techie

You have to be careful before you switch users because you need to make sure that your id had proper permissions on HDFS to read and write. Answer is yes but again, make sure you do this

su - hdfs

hdfs dfs -mkdir /user/tom

hdfs dfs -chown -R tom:hdfs /user/tom

Re: Running spark programs as spark OS user

techiegreenhorn — Fri, 05 Feb 2016 00:06:31 GMT

@Neeraj Sabharwal

The reason I ask this question is because in the tech preview tutorial, its explicitly mentioned to change to 'spark' user before running the spark-shell commands. So wondering whats the need for it.

Assume, I have logged in as user 'tom' and have a corresponding folder on hdfs - /user/tom with read and write privileges. In that case, can I run spark-shell without changing to 'spark' user as mentioned in the link?

Re: Running spark programs as spark OS user

aervits — Fri, 05 Feb 2016 00:07:24 GMT

@Greenhorn Techie answer is still yes

Re: Running spark programs as spark OS user

techiegreenhorn — Fri, 05 Feb 2016 00:09:31 GMT

Thanks @Artem Ervits for the clarification. I got confused because of the explicit mention of it in the tutorial.

Re: Running spark programs as spark OS user

aervits — Fri, 05 Feb 2016 00:11:27 GMT

sure thing @Greenhorn Techie try it out and accept best answer. Also encourage you to publish an article on doing that.