Skip to main content
wert_1311
wert_1311
Gold ⭐️
October 31, 2019
Solved

Yarn container logs

Hi

 

I am running out of space on one of our DN, and found that Yarn container logs are consuming around 50GB, though there are 3 Spark jobs/applications running which can be seen under Yarn > Applications tab, but I see a lot of other application ids also listed, would like to know if these could be deleted to reclaim space (the ones which are not running)

Secondly, any precautions to be taken into consideration while deleting these container logs. Any guidance would be of great help.

 

Regards

Wert

 

 

gsthina
gsthinaAnswer
Gold ⭐️
November 6, 2019
Hi @wert_1311,

There is an option to just stop a single NodeManager (NM) and clean that
usercache there. So, there will not be any applications affected due to
this. However, it is worth keeping in mind, even if you stop a single
NodeManager, it has some effect on the currently running jobs. The jobs
running on that NM will be stopped and will be restarted on another NM. So,
jobs will run longer than expected because the containers have to start
again somewhere else.

Hope this helps.
View in thread

8 replies

gsthina
Gold ⭐️
gsthina
Gold ⭐️
October 31, 2019

Hey, 

 

I think, deleting container logs may be a good option to save space. However, if you would like to grab the yarn logs for analysing the old jobs, then you may need those container logs. 

 

Also, I think those analyses are required, when a job fails. So, if you think those jobs will not be dug again to gather any of the historic insights, then you may feel free to clear them.

Translate
EricL
Platinum ⭐️⭐️
EricL
Platinum ⭐️⭐️
October 31, 2019
@wert_1311 ,

Can you confirm what's the location of the log that is filled up? Can you go to CM to confirm if it is really the container log directory?
Translate
wert_1311
wert_1311Author
Gold ⭐️
November 1, 2019

@gsthina Thanks for your reply, post clean-up container logs are at comfortable at 660M.

@EricL path for container logs are (/data/yarn/container-logs), since I have you guys with me on this I would also like to check something about yarn filecache.

I see it’s consuming around 9.3G of space any way we can have this reduced? via an automated way?

Following are the Yarn settings which I, think controls filecache…. However, you guys are the experts

yarn.nodemanager.localizer.cache.target-size-mb = 10GB

yarn.nodemanager.localizer.cache.cleanup.interval-ms = 10 Minutes

 

Spce Used:

du -shc /data/yarn/nm/usercache/MyApplication/*

981M    /data/yarn/nm/usercache/ MyApplication /appcache

9.3G    /data/yarn/nm/usercache/ MyApplication /filecache

Translate
gsthina
Gold ⭐️
gsthinaAnswer
Gold ⭐️
November 6, 2019
Hi @wert_1311,

There is an option to just stop a single NodeManager (NM) and clean that
usercache there. So, there will not be any applications affected due to
this. However, it is worth keeping in mind, even if you stop a single
NodeManager, it has some effect on the currently running jobs. The jobs
running on that NM will be stopped and will be restarted on another NM. So,
jobs will run longer than expected because the containers have to start
again somewhere else.

Hope this helps.
Translate
Terms & ConditionsAccessibility statement