Hi Masters, It is look like Our Livy job (Spark Job) is failing intermittenly. when i check the logs, it is failing with followign error: java.io.FileNotFoundException: File/Folder does not exist: /adf/Clusters/spark-prod/clustername/var/log/spark/apps/application_xxxx_1530_5.inprogress [fe0eda21-c40c-42db-9890-b87d8e5a0719][2018-06-13T21:22:27.5024183-07:00] But when i check the ADLS whether this transient file exists or not , but it not there and seeing these files as shown below: -rwxrwxrwx+1 sshuser sshuser3.2 M 2018-06-09 08:01 adl://home/var/log/spark/apps/application_1528497421741_0109_2 -rwxrwxrwx+1 sshuser sshuser3.0 M 2018-06-09 08:02 adl://home/var/log/spark/apps/application_1528497421741_0109_3 It is look like transient file doesn't exit. any thoughts on it . I am thinking, it might be due to insufficient resource to handle the job it is failing. Thanks, Harsha Kudumula.
... View more