yarn logs + stdout huge size

[#1]

hi all

we have issue on data-node machine ( worker machine )

we notice that /var became full and that because stdout file is 89G

example:

/var/Hadoop/yarn/log/application
183625335_0110/container_e54_1532846235_0180_02_025216/stdout

please advise what could be the reason for this issue and how to solve it so stdout will not became as huge size

[#2]

@Michael Bronson

This usually happens for long running applications such as streaming applications which are very verbose.

I suggest you check the application id and see who lunched the application and reach out to them and asked them to :

1. Reduce the amount of logging to stdout for this application

2. Is also possible to configure rolling logs on yarn - so they have this option if they need to keep verbose but at least you could restrict the size of the logs and how many you like to keep

HTH

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

[#3]

@felix about - Reduce the amount of logging to stdout for this application , how to do that?

[#4]

about - Is also possible to configure rolling logs on yarn - so they have this option if they need to keep verbose but at least you could restrict the size of the logs and how many you like to keep

since we are talking about the stdout file , we not want to roll this file , what we want is to limit the size for example max 1G