Support Questions
Find answers, ask questions, and share your expertise

Spark temporary files

Highlighted

Spark temporary files

Rising Star

Hello All,

We have Spark jobs which are run in local mode and are creating a lot of application temporary files which are stored in local /tmp folder, would need some assistance in getting these deleted automatically.

Is there some configuration setting in CM that can be used to get this fixed.

CDH & CM - 5.16.2

2 REPLIES 2
Highlighted

Re: Spark temporary files

Cloudera Employee

Hi,

 

Have you enabled log aggregation in your Configuration?  Could you please enable it if not enabled?

 

Thanks

AKR

Highlighted

Re: Spark temporary files

Rising Star

Hi @AKR 

 

Yes yarn.log-aggregation-enable is enabled.