Reply
Explorer
Posts: 10
Registered: ‎02-10-2015

Spark2 History Server failed to start in CDH5.11.0

I have recently created two node hadoop cluster with CDH5.11.0 with Cloudera manager. It installed spark 1.16 and I am able to launch spark-shell successfully. Later, I installed spark2 from CSD and getting below error when I try to start the service.

 

 

+ exec /data/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/bin/spark-class org.apache.spark.deploy.history.HistoryServer --properties-file /var/run/cloudera-scm-agent/process/75-spark2_on_yarn-SPARK2_YARN_HISTORY_SERVER/spark2-conf/spark-history-server.conf
Exception in thread "main" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:278)
	at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
Caused by: java.io.FileNotFoundException: Log directory specified does not exist: file:/user/spark/spark2ApplicationHistory
	at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:207)
	at org.apache.spark.deploy.history.FsHistoryProvider.initialize(FsHistoryProvider.scala:153)
	at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:149)
	at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:77)
	... 6 more
Caused by: java.io.FileNotFoundException: File file:/user/spark/spark2ApplicationHistory does not exist
	at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:598)
	at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:811)
	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:588)
	at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:425)
	at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:197)
	... 9 more

 

File file:/user/spark/spark2ApplicationHistory does not exist

I have verified this file exists in HDFS and has full access to any user.

Please help me if there are any solution to come out of this problem.

Explorer
Posts: 11
Registered: ‎10-04-2017

Re: Spark2 History Server failed to start in CDH5.11.0

seems you have to create the directory as per the log, pls check 

/user/spark/spark2ApplicationHistory
Highlighted
Cloudera Employee
Posts: 60
Registered: ‎11-16-2015

Re: Spark2 History Server failed to start in CDH5.11.0

The file prefix looks interesting. To me, it suggests that the SHS (Spark2-History-Server) is trying to look for a log directory which is local to the host where SHS is supposed to run (file:/) and not within HDFS (hdfs:/)

 

Caused by: java.io.FileNotFoundException: File file:/user/spark/spark2ApplicationHistory does not exist

 

Can you share the values for:

Cloudera Manager > Spark2 > Configuration > spark.eventLog.dir

 

$ cat /var/run/cloudera-scm-agent/process/75-spark2_on_yarn-SPARK2_YARN_HISTORY_SERVER/spark2-conf/spark-history-server.conf

 

$ hdfs dfs -ls /user/spark

Announcements