Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

CDH spark cannot submit job with throwing java.io.filenotfoundexception

Highlighted

CDH spark cannot submit job with throwing java.io.filenotfoundexception

Explorer

Hi, there, 

 

While using CDH spark standalone and try to submit jobs, the SparkContext would not be initiated and throw the following exception info:

 

Noting that the cloudera manager has the default config of spark.eventLog.dir with /user/spark/applicationHistory , AND it can successfully

 

create the target directory. However, when initiating spark submit, the sparkcontext would always throw such exception info. 

 

Do we need to change something or manually modifying the /etc/spark/conf/spark-defaults.conf and config the spark.eventLog.dir with the 

 

hdfs-prefix url as : hdfs://<host>:<port>/user/spark/applicationHistory ? 

 

If this is some permission issue, then it would be weird. Cause we didnot change permission of the /user/spark directory and it is left as the default.

 

Hope for some useful guidance, thanks.

 

 INFO BlockManagerMaster: Registered BlockManager
Exception in thread "main" java.io.FileNotFoundException: 
/user/spark/applicationHistory/application_1432824195832_1275.inprogress 
(No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.(FileOutputStream.java:221)
at java.io.FileOutputStream.(FileOutputStream.java:110)
at 
org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:117)
at org.apache.spark.SparkContext.(SparkContext.scala:399)
at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:28)
at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

 

 

 

2 REPLIES 2

Re: CDH spark cannot submit job with throwing java.io.filenotfoundexception

New Contributor

Did you get this resolved?  I have the exact same problem.

 

Re: CDH spark cannot submit job with throwing java.io.filenotfoundexception

New Contributor

I actually got it resolved.  I moved my Spark history server from 1 node to another and the error went away.  I then switched it back to the original node and it still worked.