Best practices to correctly handle multiple concurrently jobs on YARN with limited memory

[#1]

Hello everyone!

I have a typical scenario where there are multiple pipelines running on Oozie, each one with different dependencies and time schedules. These pipelines comprise different jobs like Hive, Spark, Java etc. Many of these jobs are heavy on memory, the cluster has a total of 840 GB of RAM, so let's say that the memory is enough to complete any of these jobs but could not be enough to allow several of these jobs to run and complete at the same time.

Sometimes happen that few of these jobs need  to run concurrently, in this case I've noticed a sort of starvation in YARN. None of the jobs continues the execution, there are a lot of heartbeats in the logs, and none eventually completes.

YARN is set to use the Fair Scheduler, I would imagine that in a situation like this it should give resources at least to one of the job but it seems that all the jobs are fighting for resources and YARN is not capable to handle the impasse.

I would like to know which are the best practices to handle these type of scenarios. Do I need to define different YARN queues with different resources/priority (actually all the jobs run on the default queue)?

[#2]

Oozie 4 and below makes this scenario special because its launcher task
containers actually act as drivers for further application launches.

Since these launcher task containers are singular map tasks, they are not
considered as Application Masters by YARN, the default defence of limiting
the AM share of memory and CPU in the pool to avoid deadlock starvation
does not work.

This results in situations where enough Oozie launcher jobs take up all of
the memory/CPU resources, leaving no room for actual job requests to pass
and start a container.

The best way to solve this would be to have your Oozie launchers go to a
different pool than the applications it runs. You can achieve this by
configuring oozie.launcher.mapreduce.job.queuename in the workflow actions
to a pool different than where the apps are intended to go to.

Another way to solve it is to use Uber mode for Oozie launchers, but this
comes with a few faults around configuring and use of native libraries,
classpath isolation, etc. and is not recommended anymore.

In Oozie 5+ the special Oozie AM application can help avoid this, as it
will be considered part of the AM share of any pool:
https://issues.apache.org/jira/browse/OOZIE-1770

[#3]

Hi @Harsh J, thank you very much for these informations (I am using Oozie server build version: 4.1.0-cdh5.13.2)!

So if I understand correctly I need to add two properties in the oozie actions configuration, one specifying the launcher queue and one specifying the job queue.

Below it is shown a sqoop action where I have added these two properties (in bold):

<action name="DLT01V_VPAXINF_IMPORT_ACTION">
   <sqoop xmlns="uri:oozie:sqoop-action:0.2">
      <job-tracker>${jobTracker}</job-tracker>
      <name-node>${nameNode}</name-node>
      <configuration>

         <property>
            <name>oozie.launcher.mapred.job.queue.name</name>
            <value>oozie_launcher_queue</value>
         </property>

         <property>
            <name>mapred.job.queue.name</name>
            <value>job_queue</value>
         </property>
         
         <property>
            <name>oozie.launcher.mapreduce.map.java.opts</name>
            <value>-Xmx4915m</value>
         </property>
         <property>
            <name>oozie.launcher.mapreduce.reduce.java.opts</name>
            <value>-Xmx9830m</value>
         </property>
         <property>
            <name>oozie.launcher.yarn.app.mapreduce.am.command-opts</name>
            <value>-Xmx4915m</value>
         </property>
      </configuration>

      [...]

   </sqoop>

   [...]
</action>

I have some questions:

1. Do I need to define the queues "oozie_launcher_queue" and "job_queue" somewhere on the CDH or can I just use them providing the names? If yes, how should I define these queues? There are recommended settings?
2. In case of a Spark action, do I still to specify the queue? If yes, with which property (since Spark does not use MapReduce)?
3. Does it make sense to specify values for oozie.launcher.mapreduce.map.java.opts, oozie.launcher.mapreduce.reduce.java.opts, oozie.launcher.yarn.app.mapreduce.am.command-opts as I did in the example? I am asking because I've noticed in the Yarn ResourceManager that the Oozie launchers take a big amount of memory (about 30 GB each), is this normal?

Thank you for the support!

[#4]

Hi @ludof,

> Do I need to define the queues "oozie_launcher_queue" and "job_queue"
somewhere on the CDH or can I just use them providing the names? If yes,
how should I define these queues? There are recommended settings?

This depends on what you'd like to control. While simply specifying
different existing (or auto-created) queues will solve your immediate
problem, you can further control the total # of jobs Oozie can run in
parallel by applying restrictions over "oozie_launcher_queue" queue
settings in FairScheduler.

> In case of a Spark action, do I still to specify the queue? If yes, with
which property (since Spark does not use MapReduce)?

The Spark "action" part is still launched by Oozie via the standard
MapReduce 1-map job launcher. The properties of oozie.launcher.* still
applies here.

> Does it make sense to specify values for
oozie.launcher.mapreduce.map.java.opts,
oozie.launcher.mapreduce.reduce.java.opts,
oozie.launcher.yarn.app.mapreduce.am.command-opts as I did in the example?

Yes it is OK to specify those (and/or the *.memory.mb variants for direct
resource request controls) to reduce the footprint of the usually-light
Oozie launcher jobs.

[#5]

Thank you very much @Harsh J!

If I got it correctly these parameters

• oozie.launcher.mapreduce.map.java.opts
• oozie.launcher.mapreduce.reduce.java.opts
• oozie.launcher.yarn.app.mapreduce.am.command-opts

control the maximum amount of memory allocated for the Oozie launcher.

What are the equivalent parameters to control the memory allocated for the action instead (e.g. a Sqoop action), as shown in the image?