Reply
New Contributor
Posts: 2
Registered: ‎12-24-2018

Getting FileNotFoundException and LeaseExpired Exception while writing a df to hdfs path

I am using spark 1.6 to read messages from Tibco.i am taking each Rdd in dstream ,performing some tranformations and converting it to a data frame and then finally writing to a hdfs path.My job runs for few batches then it starts erroring out saying _temporary file not found. And finally the job is aborted.Below is the detailed link to stack overflow question.

P.s I am already using spark.speculation=false

https://stackoverflow.com/questions/53911624/getting-java-io-filenotfoundexception-while-saving-a-da...
Cloudera Employee
Posts: 7
Registered: ‎11-21-2018

Re: Getting FileNotFoundException and LeaseExpired Exception while writing a df to hdfs path

Would you please try submitting Spark job by disabling
'spark.streaming.unpersist'
{{
--conf spark.streaming.unpersist=false
}}
Announcements