Getting FileNotFoundException and LeaseExpired Exception while writing a df to hdfs path

[#1]

I am using spark 1.6 to read messages from Tibco.i am taking each Rdd in dstream ,performing some tranformations and converting it to a data frame and then finally writing to a hdfs path.My job runs for few batches then it starts erroring out saying _temporary file not found. And finally the job is aborted.Below is the detailed link to stack overflow question.

P.s I am already using spark.speculation=false

https://stackoverflow.com/questions/53911624/getting-java-io-filenotfoundexception-while-saving-a-da...

[#2]

Would you please try submitting Spark job by disabling
'spark.streaming.unpersist'
{{
--conf spark.streaming.unpersist=false
}}