Specify different queue when using Oozie web services sqoop job proxy submission?

[#1]

We are trying to submit Oozie sqoop jobs using the Oozie Web Services API, sqoop job proxy submission as documented here:

https://oozie.apache.org/docs/4.1.0/WebServicesAPI.html#Proxy_Sqoop_Job_Submission

We are working with a kerberized cluster with Knox gateway, and have not been able to find good documentation on how to implement the AuthOozieClient in an environment like this. (If anyone has some documentation they could share on this, that would be great!)

The sqoop command is generated dynamically based on selections made by the user in a web interface, and the XML sent to Oozie. The jobs are going to the “default” queue, and we want them to go to their own queue.

I have tried adding the following to the XML that is passed to the web service, but the jobs are still being submitted to the default queue:

<property> <name>mapreduce.job.queuename</name> <value>sqoop</value> </property>

Using this property when running sqoop from the command line, or adding it to a workflow.xml file in the configuration section, allows the jobs to be submitted to the sqoop queue.

Does anyone know if this web service supports specifying properties other than the required and optional properties documented? Is there a different way to accomplish this while still using the sqoop job proxy submission?

Thank you for any help you can provide.

[#2]

I don't see an example but theres a mention of parameter called queueName, can you pass that as argument? Most likely your original job.Properties file should have that defined. So you can override in your proxy call that parameter with correct queue.

Updating coordinator definition and properties
Existing coordinator definition and properties will be replaced by new definition and properties. Refer Updating coordinator definition and properties

PUT oozie/v2/job/0000000-140414102048137-oozie-puru-C?action=update

Response:

HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
{"update":
     {"diff":"**********Job definition changes**********\n******************************************\n**********Job conf changes****************\n@@ -8,16 +8,12 @@\n
          <value>hdfs:\/\/localhost:9000\/user\/purushah\/examples\/apps\/aggregator\/coordinator.xml<\/value>\r\n   <\/property>\r\n   <property>\r\n
          -    <name>user.name<\/name>\r\n
          -    <value>purushah<\/value>\r\n
          -    <\/property>\r\n
          -  <property>\r\n     <name>start<\/name>\r\n
               <value>2010-01-01T01:00Z<\/value>\r\n   <\/property>\r\n   <property>\r\n
          -    <name>newproperty<\/name>\r\n
          -    <value>new<\/value>\r\n
          +    <name>user.name<\/name>\r\n
          +    <value>purushah<\/value>\r\n   <\/property>\r\n   <property>\r\n
               <name>queueName<\/name>\r\n******************************************\n"
      }
}

queueName = default

[#3]

when submitting jobs via oozie, there is a laucher job and the launched job (in this case mr job launched by Sqoop). You are probably seeing the launcher job getting submitted to the default queue. To get the launcher job also go to sqoop queue, you need to add the following config property to the workflow.xml or the job propert

oozie.launcher.mapreduce.job.queuename = sqoop

In general, if you want to pass any config to the launcher job, you need to prefix the config name with oozie.launcher. -- For example, if you are running a hive action and the need to configure a larger map memory for the hive client

[#4]

Thank you for your response!

I added that parameter to the XML:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<configuration>
 <property>
  <name>fs.default.name</name>
  <value> ... removed ... </value>
 </property>
 <property>
  <name>mapred.job.tracker</name>
  <value> ... removed ... </value>
 </property>
 <property>
  <name>oozie.launcher.mapreduce.job.queuename</name>
  <value>sqoop</value>
 </property>
 <property>
  <name>mapreduce.job.queuename</name>
  <value>sqoop</value>
 </property>
 <property>
  <name>user.name</name>
  <value> ... removed ... </value>
 </property>
 <property>
  <name>oozie.sqoop.command</name>
  <value> ... removed ... </value>
 </property>
 <property>
  <name>oozie.libpath</name>
  <value> ... removed ... </value>
 </property>
 <property>
  <name>oozie.proxysubmission</name>
  <value>true</value>
 </property>
</configuration>

And ran the application, but both jobs are still getting submitted to the default queue:

It is as if the web service is ignoring these options.

[#5]

You should set something like queueName = sqoop in your job properties and refer to it in the workflow action config for the two parameters. Sorry if it was not clear

[#6 · Accepted solution]

@fogartyamanda

The configuration properties will be passed to the sqoop job in following way:

    <property>
        <name>oozie.sqoop.options.1</name>
        <value>-Doozie.launcher.mapreduce.job.queuename=sqoop</value>
    </property>
    <property>
        <name>oozie.sqoop.options.0</name>
        <value>-Dmapreduce.job.queuename=sqoop</value>
    </property>    
    <property>
          <name>oozie.sqoop.options.size</name>
          <value>2</value>
    </property>

Thanks.

[#7]

Abhishek,

Thank you, this worked!

[#8]

Simply set a propert in oozie workflow ie. hadoop property "MAPRED>JOB>QUEUENAME" value would be "YOUR_QUEUE_NAME". this worked for me my oozie workflow is being submitted to a particular QUEUE now. Cheers.

,

I also had the similar requirement to submit the job in a particular queue. We simply need to change at one place that is oozie workflow settings. Add one hadoop property there as "MAPRED.JOB.QUEUENAME" and Value is "YOUR_QUEUE_NAME". By this way workflow is submitted to defined queue.