Reply
Highlighted
Explorer
Posts: 10
Registered: ‎02-10-2015

Spark2 History Server failed to start in CDH5.11.0

I have recently created two node hadoop cluster with CDH5.11.0 with Cloudera manager. It installed spark 1.16 and I am able to launch spark-shell successfully. Later, I installed spark2 from CSD and getting below error when I try to start the service.

 

 

+ exec /data/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/bin/spark-class org.apache.spark.deploy.history.HistoryServer --properties-file /var/run/cloudera-scm-agent/process/75-spark2_on_yarn-SPARK2_YARN_HISTORY_SERVER/spark2-conf/spark-history-server.conf
Exception in thread "main" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:278)
	at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
Caused by: java.io.FileNotFoundException: Log directory specified does not exist: file:/user/spark/spark2ApplicationHistory
	at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:207)
	at org.apache.spark.deploy.history.FsHistoryProvider.initialize(FsHistoryProvider.scala:153)
	at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:149)
	at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:77)
	... 6 more
Caused by: java.io.FileNotFoundException: File file:/user/spark/spark2ApplicationHistory does not exist
	at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:598)
	at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:811)
	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:588)
	at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:425)
	at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:197)
	... 9 more

 

File file:/user/spark/spark2ApplicationHistory does not exist

I have verified this file exists in HDFS and has full access to any user.

Please help me if there are any solution to come out of this problem.

New Contributor
Posts: 8
Registered: ‎10-04-2017

Re: Spark2 History Server failed to start in CDH5.11.0

seems you have to create the directory as per the log, pls check 

/user/spark/spark2ApplicationHistory
Cloudera Employee
Posts: 33
Registered: ‎11-16-2015

Re: Spark2 History Server failed to start in CDH5.11.0

The file prefix looks interesting. To me, it suggests that the SHS (Spark2-History-Server) is trying to look for a log directory which is local to the host where SHS is supposed to run (file:/) and not within HDFS (hdfs:/)

 

Caused by: java.io.FileNotFoundException: File file:/user/spark/spark2ApplicationHistory does not exist

 

Can you share the values for:

Cloudera Manager > Spark2 > Configuration > spark.eventLog.dir

 

$ cat /var/run/cloudera-scm-agent/process/75-spark2_on_yarn-SPARK2_YARN_HISTORY_SERVER/spark2-conf/spark-history-server.conf

 

$ hdfs dfs -ls /user/spark

Announcements