Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark2 History Server failed to start in CDH5.11.0

Highlighted

Spark2 History Server failed to start in CDH5.11.0

Explorer

I have recently created two node hadoop cluster with CDH5.11.0 with Cloudera manager. It installed spark 1.16 and I am able to launch spark-shell successfully. Later, I installed spark2 from CSD and getting below error when I try to start the service.

 

 

+ exec /data/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/bin/spark-class org.apache.spark.deploy.history.HistoryServer --properties-file /var/run/cloudera-scm-agent/process/75-spark2_on_yarn-SPARK2_YARN_HISTORY_SERVER/spark2-conf/spark-history-server.conf
Exception in thread "main" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:278)
	at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
Caused by: java.io.FileNotFoundException: Log directory specified does not exist: file:/user/spark/spark2ApplicationHistory
	at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:207)
	at org.apache.spark.deploy.history.FsHistoryProvider.initialize(FsHistoryProvider.scala:153)
	at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:149)
	at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:77)
	... 6 more
Caused by: java.io.FileNotFoundException: File file:/user/spark/spark2ApplicationHistory does not exist
	at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:598)
	at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:811)
	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:588)
	at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:425)
	at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:197)
	... 9 more

 

File file:/user/spark/spark2ApplicationHistory does not exist

I have verified this file exists in HDFS and has full access to any user.

Please help me if there are any solution to come out of this problem.

2 REPLIES 2

Re: Spark2 History Server failed to start in CDH5.11.0

Contributor

seems you have to create the directory as per the log, pls check 

/user/spark/spark2ApplicationHistory

Re: Spark2 History Server failed to start in CDH5.11.0

Expert Contributor

The file prefix looks interesting. To me, it suggests that the SHS (Spark2-History-Server) is trying to look for a log directory which is local to the host where SHS is supposed to run (file:/) and not within HDFS (hdfs:/)

 

Caused by: java.io.FileNotFoundException: File file:/user/spark/spark2ApplicationHistory does not exist

 

Can you share the values for:

Cloudera Manager > Spark2 > Configuration > spark.eventLog.dir

 

$ cat /var/run/cloudera-scm-agent/process/75-spark2_on_yarn-SPARK2_YARN_HISTORY_SERVER/spark2-conf/spark-history-server.conf

 

$ hdfs dfs -ls /user/spark