How can I reference Falcon feed directory in a Oozie workflow

dwee101 — Wed, 08 Jun 2016 06:23:20 GMT

I used the Falcon UI to create a input feed and output feed. Both are hdfs directories with the input directory (inputDir) populated by flume and the ouput directory (outputDir) to be populated by my oozie workflow shell-action.

Next I created a Falcon process which references my oozie workflow.xml file and I added a input with a name inputDir and output with the name outputDir.

below is the shell-action I am using in my oozie workflow.xml

        <shell xmlns="uri:oozie:shell-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
            </configuration>
            <exec>/usr/hdp/current/spark-client/bin/spark-submit</exec>
            <argument>--master</argument>
            <argument>yarn-client</argument>
            <argument>wordCountWArgs.py</argument>
            <argument>${inputDir}</argument>
            <argument>${outputDir}</argument>
            <env-var>HADOOP_USER_NAME=falcon</env-var>
            <file>${nameNode}/user/falcon/apps/scripts/wordCountWArgs.py#wordCountWArgs.py</file>   
            <capture-output/>
        </shell>

the job fails and I can see a message in the log that it couldn't reference ${inputDir} and ${outputDir}

Does anyone have any idea of where I am going wrong?

Thanks in advance.

Re: How can I reference Falcon feed directory in a Oozie workflow

cnormile — Tue, 14 Jun 2016 03:09:46 GMT

Oozie cannot directly reference Falcon objects. To use Oozie with Falcon, create an Input Feed and Output feed and a Falcon Oozie process. Falcon will then generate the Oozie workflow XML.

question How can I reference Falcon feed directory in a Oozie workflow in Archives of Support Questions (Read Only)

How can I reference Falcon feed directory in a Oozie workflow

Re: How can I reference Falcon feed directory in a Oozie workflow