__spark_conf__.zip is not found

[#1]

I am trying to submit the application through livy batches through the Postman POST call. I do see the following error in the logs. But i am able to execute the commands in the interactive livy sessions through Curl.

I have checked the following properties. Looks like they are fine.

export SPARK_CONF_DIR=/etc/spark2/conf

export HADOOP_CONF_DIR=/etc/hadoop/conf

Please suggest if i am missing any

"Diagnostics: File does not exist: hdfs://<host:port>/user/livy/.sparkStaging/application_TimeStamp/__spark_conf__.zip"

[#2]

We had this issue... What I found from logs is that the livy client copies this zip from /tmp/sparkxxxxx to HDFS:/user/<username>/.sparkStaging/....

In our case, livy user impersonation was enabled. Hence <username>

Livy wad able to create ".sparkStaging/<application-id>" directory... but that directory is owned by the <username> with 700 permission.... And then the zip file could not be copied to that directory by Livy.

I found this by a writing a small shell script that continuously spins "ls -lR"ing the HDFS /user/<username>/.sparkStaging directory.. (while true:; do; hdfs dfs -ls -R /user/<username>/.sparkStaging; done;) --> Ran this as "livy" user..

So, we just gave blanket permissions in Ranger for "livy" to "/user" recursively. And that was it! We got it working!

[#3]

We had same issue, all we did is to remove local[*] from our code, and submitted again in yarn mode and it got fixed.