Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Specify different queue when using Oozie web services sqoop job proxy submission?

avatar
New Contributor

We are trying to submit Oozie sqoop jobs using the Oozie Web Services API, sqoop job proxy submission as documented here:

https://oozie.apache.org/docs/4.1.0/WebServicesAPI.html#Proxy_Sqoop_Job_Submission

We are working with a kerberized cluster with Knox gateway, and have not been able to find good documentation on how to implement the AuthOozieClient in an environment like this. (If anyone has some documentation they could share on this, that would be great!)

The sqoop command is generated dynamically based on selections made by the user in a web interface, and the XML sent to Oozie. The jobs are going to the “default” queue, and we want them to go to their own queue.

I have tried adding the following to the XML that is passed to the web service, but the jobs are still being submitted to the default queue:

<property> <name>mapreduce.job.queuename</name> <value>sqoop</value> </property>

Using this property when running sqoop from the command line, or adding it to a workflow.xml file in the configuration section, allows the jobs to be submitted to the sqoop queue.

Does anyone know if this web service supports specifying properties other than the required and optional properties documented? Is there a different way to accomplish this while still using the sqoop job proxy submission?

Thank you for any help you can provide.

1 ACCEPTED SOLUTION

avatar
Contributor

@fogartyamanda

The configuration properties will be passed to the sqoop job in following way:

    <property>
        <name>oozie.sqoop.options.1</name>
        <value>-Doozie.launcher.mapreduce.job.queuename=sqoop</value>
    </property>
    <property>
        <name>oozie.sqoop.options.0</name>
        <value>-Dmapreduce.job.queuename=sqoop</value>
    </property>    
    <property>
          <name>oozie.sqoop.options.size</name>
          <value>2</value>
    </property>

Thanks.

View solution in original post

7 REPLIES 7

avatar
Master Mentor

I don't see an example but theres a mention of parameter called queueName, can you pass that as argument? Most likely your original job.Properties file should have that defined. So you can override in your proxy call that parameter with correct queue.

Updating coordinator definition and properties
Existing coordinator definition and properties will be replaced by new definition and properties. Refer Updating coordinator definition and properties

PUT oozie/v2/job/0000000-140414102048137-oozie-puru-C?action=update

Response:

HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
{"update":
     {"diff":"**********Job definition changes**********\n******************************************\n**********Job conf changes****************\n@@ -8,16 +8,12 @@\n
          <value>hdfs:\/\/localhost:9000\/user\/purushah\/examples\/apps\/aggregator\/coordinator.xml<\/value>\r\n   <\/property>\r\n   <property>\r\n
          -    <name>user.name<\/name>\r\n
          -    <value>purushah<\/value>\r\n
          -    <\/property>\r\n
          -  <property>\r\n     <name>start<\/name>\r\n
               <value>2010-01-01T01:00Z<\/value>\r\n   <\/property>\r\n   <property>\r\n
          -    <name>newproperty<\/name>\r\n
          -    <value>new<\/value>\r\n
          +    <name>user.name<\/name>\r\n
          +    <value>purushah<\/value>\r\n   <\/property>\r\n   <property>\r\n
               <name>queueName<\/name>\r\n******************************************\n"
      }
}
queueName = default

avatar
Expert Contributor

when submitting jobs via oozie, there is a laucher job and the launched job (in this case mr job launched by Sqoop). You are probably seeing the launcher job getting submitted to the default queue. To get the launcher job also go to sqoop queue, you need to add the following config property to the workflow.xml or the job propert

oozie.launcher.mapreduce.job.queuename = sqoop

In general, if you want to pass any config to the launcher job, you need to prefix the config name with oozie.launcher. -- For example, if you are running a hive action and the need to configure a larger map memory for the hive client

avatar
New Contributor

Thank you for your response!

I added that parameter to the XML:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<configuration>
 <property>
  <name>fs.default.name</name>
  <value> ... removed ... </value>
 </property>
 <property>
  <name>mapred.job.tracker</name>
  <value> ... removed ... </value>
 </property>
 <property>
  <name>oozie.launcher.mapreduce.job.queuename</name>
  <value>sqoop</value>
 </property>
 <property>
  <name>mapreduce.job.queuename</name>
  <value>sqoop</value>
 </property>
 <property>
  <name>user.name</name>
  <value> ... removed ... </value>
 </property>
 <property>
  <name>oozie.sqoop.command</name>
  <value> ... removed ... </value>
 </property>
 <property>
  <name>oozie.libpath</name>
  <value> ... removed ... </value>
 </property>
 <property>
  <name>oozie.proxysubmission</name>
  <value>true</value>
 </property>
</configuration>

And ran the application, but both jobs are still getting submitted to the default queue:

11442-default-queue.jpg

It is as if the web service is ignoring these options.

avatar
Expert Contributor

You should set something like queueName = sqoop in your job properties and refer to it in the workflow action config for the two parameters. Sorry if it was not clear

avatar
Contributor

@fogartyamanda

The configuration properties will be passed to the sqoop job in following way:

    <property>
        <name>oozie.sqoop.options.1</name>
        <value>-Doozie.launcher.mapreduce.job.queuename=sqoop</value>
    </property>
    <property>
        <name>oozie.sqoop.options.0</name>
        <value>-Dmapreduce.job.queuename=sqoop</value>
    </property>    
    <property>
          <name>oozie.sqoop.options.size</name>
          <value>2</value>
    </property>

Thanks.

avatar
New Contributor

Abhishek,

Thank you, this worked!

avatar
New Contributor

Simply set a propert in oozie workflow ie. hadoop property "MAPRED>JOB>QUEUENAME" value would be "YOUR_QUEUE_NAME". this worked for me my oozie workflow is being submitted to a particular QUEUE now. Cheers.

,

I also had the similar requirement to submit the job in a particular queue. We simply need to change at one place that is oozie workflow settings. Add one hadoop property there as "MAPRED.JOB.QUEUENAME" and Value is "YOUR_QUEUE_NAME". By this way workflow is submitted to defined queue.