Hello,   I am running CDH 5.12 QuickStart VM with package installation (no parcels, and no CM).   I can't get Spark to produce application logs in the designated HDFS directory, and consequently nothing is displayed by Spark History Server.  My Spark jobs run as part of an Oozie workflow, but no Spark logs are produced.   My  /etc/spark/conf/spark-defaults.conf contains:   spark.eventLog.enabled true spark.eventLog.dir hdfs:///user/spark/applicationHistory spark.history.fs.logDirectory hdfs:///user/spark/applicationHistory spark.yarn.historyServer.address http://quickstart.cloudera:18088   The HDFS log directory has the following permissions: $sudo -u hdfs hadoop fs -ls /user/spark Found 1 items drwxrwxrwt - spark spark 0 2017-09-06 13:31 /user/spark/applicationHistory The Oozie Spark Task runs on Yarn, and it is defined as: <spark xmlns="uri:oozie:spark-action:0.1"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <master>yarn</master> <mode>cluster</mode> .... </spark> The Oozie workflow runs correctly, and I can see the logs in the Yarn History Server, and in Hue's Oozie Dashboard. However the Spark History Server shows this:   History Server Event log directory: hdfs:///user/spark/applicationHistory No completed applications found! Did you specify the correct logging directory? Please verify your setting of spark.history.fs.logDirectory and whether you have the permissions to access it. It is also possible that your application did not run to completion or did not stop the SparkContext. The HDFS directory /user/spark/applicationHistory is empty.   I have looked everywhere in the documentation, specifically here: https://www.cloudera.com/documentation/enterprise/5-11-x/topics/admin_spark_history_server.html,  but I have not been able to find a solution, please help.   Thanks in advance, Alex Soto     ... View more