Reply
Expert Contributor
Posts: 61
Registered: ‎02-03-2016
Accepted Solution

Yarn Aggregate Log Retention Setting

We have CDH 5.7.2 installed alongside with Cloudera Manager 5.8.1 at our company. We have configured YARN log aggregation to be enabled and YARN log aggregation retain seconds set to 1 day. For some reason, the YARN job logs in the default HDFS directory /tmp/logs/ are not being deleted. Can anyone explain why this is?

 

BTW, we have both Hive and Spark jobs running on our cluster.

 

Thanks,

Ben

Posts: 642
Topics: 3
Kudos: 114
Solutions: 67
Registered: ‎08-16-2016

Re: Yarn Aggregate Log Retention Setting

Is /tmp/logs, and all subdirs, set to 770 and hadoop group?

Have you check for actual log files? The log directories are not removed. It may appear that the logs are lingering.

Use hdfs dfs -du -s -h /tmp/logs/ to see if there is any decrease over time or if it is just increasing?
Posts: 518
Topics: 14
Kudos: 87
Solutions: 45
Registered: ‎09-02-2016

Re: Yarn Aggregate Log Retention Setting

@benassi

 

As we know "Yarn Aggregate Log Retention" can control only YARN but /tmp/logs is not limited to YARN

 

So Can you check the YARN log date using below steps. 
CM -> Yarn -> Web UI -> Resource Manager web UI -> (it will open 8088 link) Click on Finished link (left side) -> Come down and click on 'Last' button -> Check the log date -> You should see only one day history data as you configured to 1 day

 

Note: Make sure CM-> Yarn -> Configuration -> Enable Log Aggregation = Enabled

 

Thanks

Kumar

Expert Contributor
Posts: 61
Registered: ‎02-03-2016

Re: Yarn Aggregate Log Retention Setting

To answer your questions:

 

The /tmp/logs and all subdirs are 770 and the group is hdfs. Should the group be hadoop instead? I see that the yarn user is not part of the hdfs group but is in the hadoop group.

 

The logs date back to Dec 18 and increase in size less than 1TB per day. We manually delete the logs to prevent it getting to big.

 

Thanks,

Ben

Expert Contributor
Posts: 61
Registered: ‎02-03-2016

Re: Yarn Aggregate Log Retention Setting

I did as you asked and see that the oldest finished is from Dec 18, and I see the logs in HDFS under /tmp/logs.

 

Log Aggregation is enabled.

 

Thanks,

Ben

Posts: 642
Topics: 3
Kudos: 114
Solutions: 67
Registered: ‎08-16-2016

Re: Yarn Aggregate Log Retention Setting

@benassi check who all belongs to the hadoop group. It should be hdfs, mapred, and yarn. The yarn account, as that is that the RM, NM, and JH run as, will need to have read/write access to be able to remove any old logs.
Expert Contributor
Posts: 61
Registered: ‎02-03-2016

Re: Yarn Aggregate Log Retention Setting

Thanks for mentioning the information about the hadoop group and permissions. It would seem, that after applying these settings, all is working.

 

Cheers,

Ben

zbz
New Contributor
Posts: 5
Registered: ‎10-07-2018

Re: Yarn Aggregate Log Retention Setting

Hi , my cluster is CDH 5.7.2,CM5.7.0,  and I meet the same touble.

we set  dfs.permissions.superusergroup=supergroup    ;  and we run the mapreduce application by 'hdfs' user, the hdfs file like this:

      drwxrwx---   - hdfs       supergroup          0 2018-06-05 15:01 /tmp/logs/hdfs

 

and the linux mapping of user to group is :

        hadoop:x:497:hdfs,mapred,yarn

        supergroup:x:505:hdfs,yarn

 

what should I do to resolve this problem?  thanks you very much.

 

Posts: 1,756
Kudos: 373
Solutions: 279
Registered: ‎07-31-2013

Re: Yarn Aggregate Log Retention Setting

@zbz,

The group ownership of all directories under /tmp/logs must be 'hadoop' or any group ID that's common between the 'yarn' and 'mapred' IDs. In your case you have it as supergroup, which does not have 'mapred' as its member, but is also the entirely wrong group to use - you do not want to grant HDFS superuser access to YARN service. I'd recommend removing 'yarn' from the 'supergroup' group.

This is what a normal installation should appear as:

# id -Gn mapred
mapred hadoop

# id -Gn yarn
yarn hadoop

# hadoop fs -ls -d /tmp/logs
drwxrwxrwt - mapred hadoop 0 2017-08-30 22:36 /tmp/logs

So if the 'hadoop' group is shared by your two IDs (mapred and yarn) then you may execute the below (as a HDFS superuser) to resolve the issue permanently:

hadoop fs -chgrp -R hadoop /tmp/logs
zbz
New Contributor
Posts: 5
Registered: ‎10-07-2018

Re: Yarn Aggregate Log Retention Setting

Thank you so munch!

I change the group of '/tmp/logs' to hadoop , and  restart the JobHistoryServer role, everything being OK.   

    So happy !

Announcements