Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to change Spark2 History Server log location?

Highlighted

How to change Spark2 History Server log location?

New Contributor

Spark2 History Server is writing truly massive logfiles to /var/log/spark2 (on the order of 20-30GB). I'd like to redirect these to /dev/null. How do I change the log location?

For the curious, the content of these logfiles (/var/log/spark2/spark-spark-org.apache.spark.deploy.history.HistoryServer-1-hdp001.cac.queensu.ca.out) is just purely the following:

17/07/11 11:00:07 ERROR FsHistoryProvider: Exception encountered when attempting to load application log hdfs://<somehostname>:8020/spark2-history/application_1494957845701_0008.inprogress
org.apache.hadoop.security.AccessControlException: Permission denied: user=spark, access=READ, inode="/spark2-history/application_1494957845701_0008.inprogress":zeppelin:hadoop:-rwxrwx---
    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:219)
    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
    at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1955)
    at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1939)
[snip]
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=spark, access=READ, inode="/spark2-history/application_1494957845701_0008.inprogress":zeppelin:hadoop:-rwxrwx---
    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:219)
    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
    at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1955)
    at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1939)
    at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPathAccess(FSDirectory.java:1913)
[snip]

Although fixing the underlying issue would be nice, right now I'd settle for just changing the log location to /dev/null so it doesn't constantly fill up the root partition on that machine.

1 REPLY 1
Highlighted

Re: How to change Spark2 History Server log location?

Guru

@Jeff Stafford, you can change the value of spark_log_dir from /etc/spark2/conf/spark-env.sh. Restart the Spark services after making configuration change.

export SPARK_LOG_DIR=/dev/null
Don't have an account?
Coming from Hortonworks? Activate your account here