Reply
Highlighted
New Contributor
Posts: 2
Registered: ‎05-03-2018

Spark job not creating the user folder under user cache while running the job

Hi All,

 

Our Dev is running a Spark job and it needs to create a folder under '/hadoop/data11/yarn/nm/usercache/' with the user running the job. The path should be like '/hadoop/data11/yarn/nm/usercache/user1'. But i'm getting error as 'java.lang.IllegalArgumentException: No Config File/hadoop/data11/yarn/nm/usercache/user1/appcache/application_1549376940004_4169/container_1549376940004_4169_01_000001/hdfs:/user/user1/pqrstu/config/input_abc1234.xml'. 

 

Please help.

Cloudera Employee
Posts: 59
Registered: ‎04-24-2017

Re: Spark job not creating the user folder under user cache while running the job

Hi ,

 

This looks like application is trying to find an hdfs file under yarn local directory

 

 hdfs:/user/user1/pqrstu/config/input_abc1234.xml

 

The file being create here should be just input_abc1234.xml

 

Not sure what might be casuing this. Can you please give us the exact command you are using to submit spark job.

 

Thanks
Bimal

Announcements