Reply
Explorer
Posts: 21
Registered: ‎12-23-2014

CDH spark cannot submit job with throwing java.io.filenotfoundexception

Hi, there, 

 

While using CDH spark standalone and try to submit jobs, the SparkContext would not be initiated and throw the following exception info:

 

Noting that the cloudera manager has the default config of spark.eventLog.dir with /user/spark/applicationHistory , AND it can successfully

 

create the target directory. However, when initiating spark submit, the sparkcontext would always throw such exception info. 

 

Do we need to change something or manually modifying the /etc/spark/conf/spark-defaults.conf and config the spark.eventLog.dir with the 

 

hdfs-prefix url as : hdfs://<host>:<port>/user/spark/applicationHistory ? 

 

If this is some permission issue, then it would be weird. Cause we didnot change permission of the /user/spark directory and it is left as the default.

 

Hope for some useful guidance, thanks.

 

 INFO BlockManagerMaster: Registered BlockManager
Exception in thread "main" java.io.FileNotFoundException: 
/user/spark/applicationHistory/application_1432824195832_1275.inprogress 
(No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.(FileOutputStream.java:221)
at java.io.FileOutputStream.(FileOutputStream.java:110)
at 
org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:117)
at org.apache.spark.SparkContext.(SparkContext.scala:399)
at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:28)
at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

 

 

 

Highlighted
New Contributor
Posts: 3
Registered: ‎10-06-2015

Re: CDH spark cannot submit job with throwing java.io.filenotfoundexception

Did you get this resolved?  I have the exact same problem.

 

New Contributor
Posts: 3
Registered: ‎10-06-2015

Re: CDH spark cannot submit job with throwing java.io.filenotfoundexception

I actually got it resolved.  I moved my Spark history server from 1 node to another and the error went away.  I then switched it back to the original node and it still worked.