Support Questions
Find answers, ask questions, and share your expertise

Run flume job from oozie shell action

Run flume job from oozie shell action

Explorer

Hi,

I want to run the flume job using shell action in oozie. I run the flume job manually and it was executed successfully.

I have created below shell script which initiate the flume job:

script.sh:

/usr/hdp/2.x.x.x-xxx/flume/bin/flume-ng agent -n agent1 --conf /etc/flume/conf -f flume.conf

-Dflume.root.logger=DEBUG,console

Flume job executed using shell script(.sh file) But while executing flume job from oozie , shell action get stuck in PREP mode.

job.xml :

<configuration>

<property> <name>nameNode</name> <value>hdfs://hostname:8020</value> </property>

<property> <name>resourceManager</name> <value>hdfs://hostname:8050</value> </property>

<property> <name>oozie.use.system.libpath</name> <value>true</value> </property>

<property> <name>oozie.wf.application.path</name> <value>hdfs://hostname:8020/oozie</value> </property>

<property> <name>queueName</name> <value>default</value> </property>

</configuration>

workflow.xml :

<workflow-app xmlns="uri:oozie:workflow:0.4" name="shell-wf">

<start to="shell-node"/> <action name="shell-node">

<shell xmlns="uri:oozie:shell-action:0.2">

<job-tracker>${resourceManager}</job-tracker>

<name-node>${nameNode}</name-node>

<configuration>

<property> <name>mapred.job.queue.name</name> <value>${queueName}</value> </property>

</configuration>

<exec>script.sh</exec>

<file>hdfs://hostname:8020/oozie/script.sh#script.sh</file>

<capture-output/>

</shell>

<ok to="end"/>

<error to="fail"/>

</action>

<kill name="fail">

<message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>

</kill>

<kill name="fail-output">

<message>Incorrect output, expected [Hello Oozie] but was [${wf:actionData('shell-node')['my_output']}]</message>

</kill>

<end name="end"/>

</workflow-app>

Thanks,

Don't have an account?