question Re: What type of disk (RAID 1, RAID 0, etc) should be used for the following yarn directories: in Archives of Support Questions (Read Only)

What type of disk (RAID 1, RAID 0, etc) should be used for the following yarn directories:

mcarrillo — Wed, 07 Oct 2015 20:01:23 GMT

Re: What type of disk (RAID 1, RAID 0, etc) should be used for the following yarn directories:

mcarrillo — Wed, 07 Oct 2015 20:04:02 GMT

forgot to add the directories:

yarn.nodemanager.log-dirs
yarn.timeline-service.leveldb-timeline-store.path
yarn.timeline-service.leveldb-state-store-path

Re: What type of disk (RAID 1, RAID 0, etc) should be used for the following yarn directories:

drice1 — Thu, 08 Oct 2015 08:52:37 GMT

In short Mirror them RAID 1 or RAID 1/0...some more details below

Hey Martin YARN Timeline Service (YTS) would fall in the category of a management service which would reside on a management node, which would probably be on the same spindles as a partition off the OS drives and should be setup as mirrored pair RAID 1 or RAID 1/0. The YARN Timeline Server is actually a very important component as it keeps current and historical information about applications run on the cluster, thus the underlying levelDB storage location is just as important. Node managers logs should be on a partition of the OS drives which as well would be mirrored....mirroring drives for OS on both Data Nodes and Management nodes is pretty common...not mandatory on Data Nodes but one of those things where the cost delta for just mirroring the OS on the Data Nodes is not huge, so might as well just mirror them.

Re: What type of disk (RAID 1, RAID 0, etc) should be used for the following yarn directories:

andrewg — Thu, 08 Oct 2015 21:58:14 GMT

If you have an SSD in the node (or, oftentimes, a RAID 1 mirror) and it's large enough, YTS database would be a good candidate for putting on it.

Re: What type of disk (RAID 1, RAID 0, etc) should be used for the following yarn directories:

mcarrillo — Thu, 08 Oct 2015 23:18:16 GMT

Great stuff Dan! Thanks for your help here.

Re: What type of disk (RAID 1, RAID 0, etc) should be used for the following yarn directories:

amcbarnett — Sat, 13 Feb 2016 11:45:44 GMT

@mcarillo the yarn.nodemanager.log-dirs is on the same mounts as your hadoop data directories. See

https://community.hortonworks.com/articles/1888/apache-tez-tuning-tips-solving-the-could-not-find.html