Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Is hdfs:///user/spark/applicationHistory hardcoded into spark-history-server 1.2.0+cdh5.3.3

Is hdfs:///user/spark/applicationHistory hardcoded into spark-history-server 1.2.0+cdh5.3.3

Expert Contributor

Just installed spark  1.2.0+cdh5.3.3

And can't start spark-history-server:

 

15/04/22 14:13:57 INFO history.HistoryServer: Registered signal handlers for [TERM, HUP, INT]
15/04/22 14:13:57 INFO spark.SecurityManager: Changing view acls to: spark
15/04/22 14:13:57 INFO spark.SecurityManager: Changing modify acls to: spark
15/04/22 14:13:57 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); users with modify permissions: Set(spark)
15/04/22 14:13:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:184)
at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
Caused by: java.io.IOException: Incomplete HDFS URI, no host: hdfs:///user/spark/applicationHistory
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:141)

 

How can I specify local path for history-server?

I don't have any HDFS. I run spark against Datastax DSE?

2 REPLIES 2
Highlighted

Re: Is hdfs:///user/spark/applicationHistory hardcoded into spark-history-server 1.2.0+cdh5.3.3

Master Collaborator

You can change the directory of course; it's just a bit of configuration you can manage in CM. Try setting it to a location without scheme? I suspect it may be expected to be an HDFS URI by Spark or CM though. Try prefixing with "file:/..." or whatever scheme you want?

Highlighted

Re: Is hdfs:///user/spark/applicationHistory hardcoded into spark-history-server 1.2.0+cdh5.3.3

Expert Contributor
Thanks for the reply, I don't use cloudera manager. I just fixed "SPARK_HISTORY_OPTS" and it works now.

export SPARK_HISTORY_OPTS="-Dspark.history.fs.logDirectory={{spark.eventLog}} -Dspark.eventLog.enabled=true"
Don't have an account?
Coming from Hortonworks? Activate your account here