Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Run flume job from oozie shell action

Highlighted

Run flume job from oozie shell action

New Contributor

Hi,

I want to run the flume job using shell action in oozie. I run the flume job manually and it was executed successfully.

I have created below shell script which initiate the flume job:

script.sh:

/usr/hdp/2.x.x.x-xxx/flume/bin/flume-ng agent -n agent1 --conf /etc/flume/conf -f flume.conf

-Dflume.root.logger=DEBUG,console

Flume job executed using shell script(.sh file) But while executing flume job from oozie , shell action get stuck in PREP mode.

job.xml :

<configuration>

<property> <name>nameNode</name> <value>hdfs://hostname:8020</value> </property>

<property> <name>resourceManager</name> <value>hdfs://hostname:8050</value> </property>

<property> <name>oozie.use.system.libpath</name> <value>true</value> </property>

<property> <name>oozie.wf.application.path</name> <value>hdfs://hostname:8020/oozie</value> </property>

<property> <name>queueName</name> <value>default</value> </property>

</configuration>

workflow.xml :

<workflow-app xmlns="uri:oozie:workflow:0.4" name="shell-wf">

<start to="shell-node"/> <action name="shell-node">

<shell xmlns="uri:oozie:shell-action:0.2">

<job-tracker>${resourceManager}</job-tracker>

<name-node>${nameNode}</name-node>

<configuration>

<property> <name>mapred.job.queue.name</name> <value>${queueName}</value> </property>

</configuration>

<exec>script.sh</exec>

<file>hdfs://hostname:8020/oozie/script.sh#script.sh</file>

<capture-output/>

</shell>

<ok to="end"/>

<error to="fail"/>

</action>

<kill name="fail">

<message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>

</kill>

<kill name="fail-output">

<message>Incorrect output, expected [Hello Oozie] but was [${wf:actionData('shell-node')['my_output']}]</message>

</kill>

<end name="end"/>

</workflow-app>

Thanks,