Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Passing Paramter to Oozie workflow

avatar
Explorer

Hi,

 

I am new Oozie and some doubts regarding the creating workflows.

 

I have java mapreduce progrm which expects 3 parameters to be passed: input directory, output directory and the field position of the record in the file that I need to pass to the program as parameter.

 

hadoop jar /home/cloudera/ooziesample.jar /user/cloudera/wordcount/input/ /user/cloudera/wordcount/output 2

 

Now I want to create a workflow for running this jar as mapreduce actions. I want to pass these 3 parameters via oozie workflow : input directory, output directory and the field position of the record. Could you please help me know how to do the same

1 ACCEPTED SOLUTION

avatar
Mentor
> is not getting written to job.properties file

The job.properties file is just for WF variable resolution and some Oozie submission control properties. They aren't used for actual job configuration, however.

> Where should define parameter and its value in Hue UI for a workflow?

Within your Action's configuration view, look for "Properties" to see a Key=Value like field. Enter your configuration properties here.

If you are using new-API values, you may want to read the requirements also at https://cwiki.apache.org/confluence/display/OOZIE/Map+Reduce+Cookbook (search page for "new-api").

> Do I need to upload .jar in lib directory of the workflow and what is the need of that?

You will need to upload the application and any non-hadoop dependencies, because Oozie runs a remote launcher job to run your classes, and the jars need to exist within the workflow lib directory for Oozie to ship them properly via distributed-cache.

Does this help?

View solution in original post

4 REPLIES 4

avatar
Mentor
You can use the new Java driver feature of the Oozie MR action to run your jar in an easier way. Follow this: http://archive.cloudera.com/cdh5/cdh/5/oozie/WorkflowFunctionalSpec.html#a3.2.2.2_Configuring_the_Ma...

avatar
Explorer

Hi,

 

Thanks for the reply.

 

 

I am trying to use Hue for creating and running the Oozie workflows.

Whatever Properties that I am trying to pass (eg: mapreduce.map.class and mapreduce.reduce.class) is not getting written to job.properties file. Hence could see those in workflow.xml as well.

 

I have few Qs:

Where should define parameter and its value in Hue UI for a workflow?

Do I need to upload .jar in lib directory of the workflow and what is the need of that?

 

My Use-case:

I want to run the java mapreduce program using mapreduce action.

I have a jar with multiple Mapper and Reducer Class defined and I want to use Oozie as driver for my Mapper and Reducer programs. And at the same time passing argument values through Oozie to my Mappers and Reducers.

 

Could you please let me know how to achieve this using Hue and Oozie?

 

Thanks

 

avatar
Mentor
> is not getting written to job.properties file

The job.properties file is just for WF variable resolution and some Oozie submission control properties. They aren't used for actual job configuration, however.

> Where should define parameter and its value in Hue UI for a workflow?

Within your Action's configuration view, look for "Properties" to see a Key=Value like field. Enter your configuration properties here.

If you are using new-API values, you may want to read the requirements also at https://cwiki.apache.org/confluence/display/OOZIE/Map+Reduce+Cookbook (search page for "new-api").

> Do I need to upload .jar in lib directory of the workflow and what is the need of that?

You will need to upload the application and any non-hadoop dependencies, because Oozie runs a remote launcher job to run your classes, and the jars need to exist within the workflow lib directory for Oozie to ship them properly via distributed-cache.

Does this help?

avatar
Explorer

Hi,

 

Your reply does help. Thanks.

 

But I still have doubts regarding passing parameters via Hue UI.

1.If I pass the parameter value using the Properties field, then will it be updated in the workflow under <agr> tag or <property> tag?

2. What is the use of Parameter field under Workflow Settings.

 

Also could you please give some insight regarding my use-case: I want to run the java mapreduce program using mapreduce action.

I have a jar with multiple Mapper and Reducer Class defined and I want to use Oozie as driver for my Mapper and Reducer programs. And at the same time passing argument values through Oozie to my Mappers and Reducers.

 

How can I achieve this using Hue UI to create and workflow?