question Re: Yarn Timeline db consuming 466GB space in Support Questions

Yarn Timeline db consuming 466GB space

anshul_sisodia — Thu, 21 Jul 2016 17:16:46 GMT

I have a scenario where yarn timeline store db is increasing day by day. In April it was 346GB and now it increased to 466GB and occupying lot of space in /var/opt/hadoop/yarn/timeline.

# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda2 30G 11G 18G 37% / udev 126G 260K 126G 1% /dev tmpfs 126G 0 126G 0% /dev/shm /dev/sda5 5.0G 2.5G 2.2G 54% /var /dev/sda7 756G 698G 21G 98% /var/opt/ /dev/sdb 2.5T 6.6G 2.3T 1% /data

when i checked disk usage on /var/opt/hadoop/yarn/timeline

du -sh * 466G leveldb-timeline-store.ldb 40K timeline-state-store.ldb

I don't know why it is occupying this much of space. What steps do i need to take to make it consume less space.

Re: Yarn Timeline db consuming 466GB space

bleonhardi — Thu, 21 Jul 2016 18:01:41 GMT

Yarn timeline store should cleanup old values. Parameters are:

Cleanup cycle ( when he deletes )

yarn.timeline-service.leveldb-timeline-store.ttl-interval-ms

Time to live ( what to delete )

yarn.timeline-service.ttl-ms

Oh and finally enable the age off

yarn.timeline-service.ttl-enable

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_yarn_resource_mgt/content/ref-e54bc3f2-f1bc-4bc6-b6cb-e6337589feb6.1.html

I would check those. If you want to clean things you can set that to a low settings and restart. Alternatively you should be able to simply delete the database if you want to its just log information after all. Finally if the parameters are correct and do not work you might want to open a support ticket.

Re: Yarn Timeline db consuming 466GB space

anshul_sisodia — Wed, 27 Jul 2016 19:43:25 GMT

Thanks It worked. I just want to know if there are any consequences after deleting the leveldb-timeline-store.ldb directory. After deleteing content of this directory and restarting timeline server, the contents of this directory regenerated and freed up the space.

We set “yarn.timeline-service.ttl-ms” from 267840000000(~443weeks) to 604800000(7 Days) to limit the size of the leveldb storage, according to hortonworks document.

Re: Yarn Timeline db consuming 466GB space

bleonhardi — Wed, 27 Jul 2016 19:58:48 GMT

The main consequences are for running jobs some of which may depend on ats ( too late for that ) and any investigation of performance of old jobs ( which are now gone ) apart from that nothing I would know about. Would be interested to know who set the retention period to 8 years 🙂 That doesnt make any sense at all. You could have simply changed that as well he should have cleaned up the data soon as well. Hope that works.

Re: Yarn Timeline db consuming 466GB space

anshul_sisodia — Wed, 27 Jul 2016 21:37:23 GMT

I made sure that no jobs were running at that time when the deletion happened :). Also I am not sure who set retention period to 8 years, have to check that.

One thing i want to know here, if before deletion, I have would have set the property to 7days and restarted only timeline server, would it remove all the older logs from leveldb and keep logs which are generated in last 7 days? I guess after restarting Yarn, Job history server also resets.. and gets clears at regular intervals.. not sure though!!

Re: Yarn Timeline db consuming 466GB space

bleonhardi — Thu, 28 Jul 2016 23:20:21 GMT

yeah if you set it to 7 days he should just start cleaning older values after restart ( potentially after hitting the clean period the interval_ms thing)