Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark on Yarn FileNotFoundException

Spark on Yarn FileNotFoundException

New Contributor

Hi,

I am using HDP 2.6.3 and spark 1.6.3.

When I ran a spark job on yarn, sometimes FileNotFoundException occurred as below, but sometimes my spark job succeeded.

I have no clue why it happens that the spark job failed sometimes, and succeeded sometimes.

diagnostics: Application application_1525840386596_0054 failed 5 times due to AM Container for appattempt_1525840386596_0054_000005 exited with  exitCode: -1000
10-05-2018 06:37:26 KST consolidate-parquet-files INFO - For more detailed output, check the application tracking page: <a href="http:/xxxx:8088/cluster/app/application_1525840386596_0054">http://xxx:8088/cluster/app/application_1525840386596_0054</a> Then click on links to logs of each attempt.
10-05-2018 06:37:26 KST consolidate-parquet-files INFO - Diagnostics: File does not exist: hdfs://emb/user/kafka/.sparkStaging/application_1525840386596_0054/__spark_conf__830919243772818200.zip
10-05-2018 06:37:26 KST consolidate-parquet-files INFO - java.io.FileNotFoundException: File does not exist: hdfs://emb/user/kafka/.sparkStaging/application_1525840386596_0054/__spark_conf__830919243772818200.zip
10-05-2018 06:37:26 KST consolidate-parquet-files INFO - 	at org.apache.hadoop.hdfs.DistributedFileSystem$26.doCall(DistributedFileSystem.java:1446)
10-05-2018 06:37:26 KST consolidate-parquet-files INFO - 	at org.apache.hadoop.hdfs.DistributedFileSystem$26.doCall(DistributedFileSystem.java:1438)
Don't have an account?
Coming from Hortonworks? Activate your account here