Member since
06-03-2015
2
Posts
0
Kudos Received
0
Solutions
06-03-2015
02:35 PM
Hi everyone, everytime our data comes and new updates occur in our cluster, an undesirable file is being created in all workers' directories.In order to cleanup automatically I changed the variable value Spark (Standalone) Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh in Gateway Default Group->Advanced Settings as : export SPARK_WORKER_OPTS="-Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.interval=60 -Dspark.worker.cleanup.appDataTtl=60" by using cloudera manager.After i make the cluster restart, it makes change in spark/conf/spark-env.sh but it does not make cleanup.Does anyone know where the mistake is or another way of cleaning up automatically ? i am using CDH 4 and Spark 1.2.2 in the cluster.
... View more
Labels: