Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark job not creating the user folder under user cache while running the job

Highlighted

Spark job not creating the user folder under user cache while running the job

New Contributor

Hi All,

 

Our Dev is running a Spark job and it needs to create a folder under '/hadoop/data11/yarn/nm/usercache/' with the user running the job. The path should be like '/hadoop/data11/yarn/nm/usercache/user1'. But i'm getting error as 'java.lang.IllegalArgumentException: No Config File/hadoop/data11/yarn/nm/usercache/user1/appcache/application_1549376940004_4169/container_1549376940004_4169_01_000001/hdfs:/user/user1/pqrstu/config/input_abc1234.xml'. 

 

Please help.

1 REPLY 1

Re: Spark job not creating the user folder under user cache while running the job

Cloudera Employee

Hi ,

 

This looks like application is trying to find an hdfs file under yarn local directory

 

 hdfs:/user/user1/pqrstu/config/input_abc1234.xml

 

The file being create here should be just input_abc1234.xml

 

Not sure what might be casuing this. Can you please give us the exact command you are using to submit spark job.

 

Thanks
Bimal