Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Apache Spark is not deleting the folders in the temporary directory (spark.local.dir)

Apache Spark is not deleting the folders in the temporary directory (spark.local.dir)

New Contributor

Hi,

Background

We are running a local mode Spark application that runs a spark job every 5 minutes using a singleton SparkContext.

- We are using Spark 1.6.2.2.4.3.0-227

- The application is long running and "sleeps" in-between the jobs

- We are using SparkContext.getOrCreate

- We are running spark in "local[*]" mode

- "spark.worker.cleanup.enabled" is set to "true"

- The application is written in Scala

- On failure we are invoking the Spark Context "stop" method in order to get a healthy SparkContext for the next job.

Problem

The "spark.local.dir" directory is filling up over time, and we eventually get "java.io.IOException: No space left on device".


-----------------

We found an old Jira ticket mentioning the issue (https://issues.apache.org/jira/browse/SPARK-7439), but it seems it was closed with the motivation "the dirs should already be cleaned up on JVM exit".

2 REPLIES 2

Re: Apache Spark is not deleting the folders in the temporary directory (spark.local.dir)

Expert Contributor

Are we closing the spark context here ? Usually a ".close()" call is done, the JVM should be able to clean up those directories .

Highlighted

Re: Apache Spark is not deleting the folders in the temporary directory (spark.local.dir)

New Contributor

Hi Sumesh,
We are using a singleton spark context (SparkContext.getOrCreate). When the business logic fails we call ".stop()" (close is not available) on it to make sure a new one is created for the next run.

Don't have an account?
Coming from Hortonworks? Activate your account here