Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

/atsv2 large size. How to purge old data?

avatar
Explorer

Hello,

I see the path /atsv2 in HDFS have large size, and it growing, this path contain embedded hbase data of Yarn ATS.


Have Anyone can explain about this path, and how to purge old data?


Thanks.

1 ACCEPTED SOLUTION

avatar
Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
4 REPLIES 4

avatar
Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Explorer

Hi Tomas,


Many thanks. I will try to do follow your guide.

avatar
Contributor

Great, @son trinh, let me know how it goes.


Oh, and also, if you don't get the amount of disk space back you need, we can set TTLs to the other data.

My recommendation above is for the metrics column families on the tables (like how much memory and CPU per container) which are the least important and also the ones that come with an expiration period by default, so that you don't lose job execution metadata (like where and what and when was executed, exit status, etc.), but if required and you are OK with not having that information after the retention period, we could also get the rest of the ATSv2 data to expire with:

alter 'prod.timelineservice.application', {NAME=> 'c',TTL => 1296000}
alter 'prod.timelineservice.application', {NAME=> 'i',TTL => 1296000}
alter 'prod.timelineservice.app_flow', {NAME=> 'm',TTL => 1296000}
alter 'prod.timelineservice.entity', {NAME=> 'c',TTL => 1296000}
alter 'prod.timelineservice.entity', {NAME=> 'i',TTL => 1296000}
alter 'prod.timelineservice.flowrun', {NAME=> 'i',TTL => 1296000}
alter 'prod.timelineservice.flowactivity', {NAME=> 'i',TTL => 1296000}
alter 'prod.timelineservice.subapplication', {NAME=> 'c',TTL => 1296000}
alter 'prod.timelineservice.subapplication', {NAME=> 'i',TTL => 1296000}

Regards,

--

Tomas

avatar
Explorer

Hi Tomas,


I applied these and run compaction manually. The size of /atsv2 is smaller.


Many thanks!