question Re: Remove directory from yarn.nodemanager.local-dirs - CDH 6.3 in Support Questions

Remove directory from yarn.nodemanager.local-dirs - CDH 6.3

mig_aguir — Sat, 09 Jan 2021 07:58:48 GMT

We made a mistake while configuring the storage of our worker nodes and we ended up adding a data mount point of a really small disk to the yarn.nodemanager.local-dirs property. This causes some of our very data-intensive spark jobs to fail as this disk runs out of space pretty quickly as these jobs generate a lot of staging data. Right now the value of this property is the following:

/u01/hadoop/yarn/nm,/u02/hadoop/yarn/nm,/u03/hadoop/yarn/nm,/u04/hadoop/yarn/nm

We just want to remove /u04/hadoop/yarn/nm from the property, all the other directories are mounted on disks with enough capacity.

So, is it safe to just remove this directory from the yarn.nodemanager.local-dirs property? Or do we have to back up the data in /u04/hadoop/yarn/nm and create another data mount point using a disk with enough capacity and put that backup there?

Re: Remove directory from yarn.nodemanager.local-dirs - CDH 6.3

GangWar — Sat, 09 Jan 2021 14:21:19 GMT

@mig_aguir From the given yarn-site.xml file "yarn.nodemanager.local-dirs" You can change above property to point to new mount directory and post restart of Yarn service new application job will use this directory for storing.

For safe side you can take a backup and store locally for time being if something goes wrong.

Re: Remove directory from yarn.nodemanager.local-dirs - CDH 6.3

mig_aguir — Sat, 09 Jan 2021 23:57:29 GMT

Cool, thanks for the help! Just another question, if I just were to remove the directory I don't want to use for storing staging data and then restarted Yarn service, would there be any problem? The "yarn.nodemanager.local-dirs" property already includes three other directories for storing so it shouldn't be a problem, right?

Re: Remove directory from yarn.nodemanager.local-dirs - CDH 6.3

GangWar — Sun, 10 Jan 2021 04:14:09 GMT

@mig_aguir Yes, I agree with you.