Spark job not creating the user folder under user cache while running the job

[#1]

Hi All,

Our Dev is running a Spark job and it needs to create a folder under '/hadoop/data11/yarn/nm/usercache/' with the user running the job. The path should be like '/hadoop/data11/yarn/nm/usercache/user1'. But i'm getting error as 'java.lang.IllegalArgumentException: No Config File/hadoop/data11/yarn/nm/usercache/user1/appcache/application_1549376940004_4169/container_1549376940004_4169_01_000001/hdfs:/user/user1/pqrstu/config/input_abc1234.xml'. 

Please help.

[#2]

Hi ,

This looks like application is trying to find an hdfs file under yarn local directory

 hdfs:/user/user1/pqrstu/config/input_abc1234.xml

The file being create here should be just input_abc1234.xml

Not sure what might be casuing this. Can you please give us the exact command you are using to submit spark job.

Thanks
Bimal