Using HDP 2.3 sandbox otb essentially and am running Spark jobs but no output in the history server:-
Did you specify the correct logging directory? Please verify your setting of spark.history.fs.logDirectory and whether you have the permissions to access it. It is also possible that your application did not run to completion or did not stop the SparkContext.
So the job ran to completion successfully (in YARN: State: Finished; Final Status: Succeeded)
Via Ambari, (based on some other info found), i updated the Spark config; adding properties to "Custom spark-defaults", which on the restart of Spark updated "/etc/spark/126.96.36.199-2950/0/spark-defaults.conf" with the following parameters:-
The /tmp/spark-events directory was already created:-
drwxrwxrwx - spark hdfs 0 2016-05-11 09:21 /tmp/spark-events
I re-ran the Spark job but still nothing in the UI.
You need to check the permissions on the application log file in hdfs://tmp/spark-events. This file must be readable by user spark
I attached the spark history-server out log.
1. Does someone have their working spark-defaults.conf settings they can share (for HDP 2.3 sandbox)?
2. Nothing is getting written to my /spark-history directory in hdfs (as per spark.history.fs.logDirectory hdfs://sandbox.hortonworks.com:50070/spark-history) (I changed it from /tmp/spark-events in my original post) ... so there is nothing to retrieve from this location to display in the history server console. What controls this part of the process?
Hi @g_neha , as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.