Spark temporary files

[#1]

Hello All,

We have Spark jobs which are run in local mode and are creating a lot of application temporary files which are stored in local /tmp folder, would need some assistance in getting these deleted automatically.

Is there some configuration setting in CM that can be used to get this fixed.

CDH & CM - 5.16.2

[#2]

Hi,

Have you enabled log aggregation in your Configuration?  Could you please enable it if not enabled?

Thanks

AKR

[#3]

Hi @AKR 

Yes yarn.log-aggregation-enable is enabled.