I am trying to submit the application through livy batches through the Postman POST call. I do see the following error in the logs. But i am able to execute the commands in the interactive livy sessions through Curl.
I have checked the following properties. Looks like they are fine.
Please suggest if i am missing any
"Diagnostics: File does not exist: hdfs://<host:port>/user/livy/.sparkStaging/application_TimeStamp/__spark_conf__.zip"
We had this issue... What I found from logs is that the livy client copies this zip from /tmp/sparkxxxxx to HDFS:/user/<username>/.sparkStaging/....
In our case, livy user impersonation was enabled. Hence <username>
Livy wad able to create ".sparkStaging/<application-id>" directory... but that directory is owned by the <username> with 700 permission.... And then the zip file could not be copied to that directory by Livy.
I found this by a writing a small shell script that continuously spins "ls -lR"ing the HDFS /user/<username>/.sparkStaging directory.. (while true:; do; hdfs dfs -ls -R /user/<username>/.sparkStaging; done;) --> Ran this as "livy" user..
So, we just gave blanket permissions in Ranger for "livy" to "/user" recursively. And that was it! We got it working!