Support Questions

Find answers, ask questions, and share your expertise

How to change Spark2 History Server log location?

New Contributor

Spark2 History Server is writing truly massive logfiles to /var/log/spark2 (on the order of 20-30GB). I'd like to redirect these to /dev/null. How do I change the log location?

For the curious, the content of these logfiles (/var/log/spark2/spark-spark-org.apache.spark.deploy.history.HistoryServer-1-hdp001.cac.queensu.ca.out) is just purely the following:

17/07/11 11:00:07 ERROR FsHistoryProvider: Exception encountered when attempting to load application log hdfs://<somehostname>:8020/spark2-history/application_1494957845701_0008.inprogress
org.apache.hadoop.security.AccessControlException: Permission denied: user=spark, access=READ, inode="/spark2-history/application_1494957845701_0008.inprogress":zeppelin:hadoop:-rwxrwx---
    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:219)
    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
    at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1955)
    at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1939)
[snip]
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=spark, access=READ, inode="/spark2-history/application_1494957845701_0008.inprogress":zeppelin:hadoop:-rwxrwx---
    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:219)
    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
    at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1955)
    at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1939)
    at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPathAccess(FSDirectory.java:1913)
[snip]

Although fixing the underlying issue would be nice, right now I'd settle for just changing the log location to /dev/null so it doesn't constantly fill up the root partition on that machine.

1 REPLY 1

Guru

@Jeff Stafford, you can change the value of spark_log_dir from /etc/spark2/conf/spark-env.sh. Restart the Spark services after making configuration change.

export SPARK_LOG_DIR=/dev/null