Support Questions
Find answers, ask questions, and share your expertise

Spark job not creating the user folder under user cache while running the job

Explorer

Hi All,

 

Our Dev is running a Spark job and it needs to create a folder under '/hadoop/data11/yarn/nm/usercache/' with the user running the job. The path should be like '/hadoop/data11/yarn/nm/usercache/user1'. But i'm getting error as 'java.lang.IllegalArgumentException: No Config File/hadoop/data11/yarn/nm/usercache/user1/appcache/application_1549376940004_4169/container_1549376940004_4169_01_000001/hdfs:/user/user1/pqrstu/config/input_abc1234.xml'. 

 

Please help.

1 REPLY 1

Re: Spark job not creating the user folder under user cache while running the job

Cloudera Employee

Hi ,

 

This looks like application is trying to find an hdfs file under yarn local directory

 

 hdfs:/user/user1/pqrstu/config/input_abc1234.xml

 

The file being create here should be just input_abc1234.xml

 

Not sure what might be casuing this. Can you please give us the exact command you are using to submit spark job.

 

Thanks
Bimal