question Re: Spark 2 in zeppelin cannot access local files due to permissions in Archives of Support Questions (Read Only)

Spark 2 in zeppelin cannot access local files due to permissions

arnaud1 — Sat, 28 Apr 2018 23:37:04 GMT

Hi,

I'm using spark2 in Zeppelin. (HDP 2.6.4)
I'm loading a csv file with spark.read

val test = spark.read      .option("header", "false")  .option("delimiter", ",")  .option("inferSchema", "true")  .csv("file:///root/share/test.csv")

/root/share/ is a shared folder from host to docker container (virtualbox).
it has privileges like dwxrwxr---

Because of the shared folder, I cannot chmod 777 on it (or maybe someone knows how to do this...)

Anyways, spark comes back with a permission denied on this forlder/file.

Questions:

1. when running such a spark2 request in zeppelin, which permissions are applied?
2. How can I "elevate" these permissions to root (so that access can be made to any folder)

Thnaks a lot for your help!

Re: Spark 2 in zeppelin cannot access local files due to permissions

falbani — Mon, 30 Apr 2018 20:57:12 GMT

@Arnaud Bohelay

Answers inline:

1. when running such a spark2 request in zeppelin, which permissions are applied?

Answer > The application will run as zeppelin user, so all access will be performed as zeppelin

2. How can I "elevate" these permissions to root (so that access can be made to any folder)

Answer > This is generally not a good idea. I suggest you create a folder with proper access permissions somewhere else. Maybe under opt. Like this:

mkdir /opt/share

chmod 777 /opt/share

and use that one instead of /root which is the home directory for the root user.

HTH

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

Re: Spark 2 in zeppelin cannot access local files due to permissions

falbani — Thu, 03 May 2018 20:27:36 GMT

@Arnaud Bohelay Please let me know the above has helped answer your question. Thanks.