Support Questions
Find answers, ask questions, and share your expertise

What is the correct way to capture logs from the Oozie Distcp action?

New Contributor

I am using the distcp Oozie action in order to replicate from source to target cluster. In doing so, I would like to capture the logs from the distcp action and write them out to the log file being written to by the rest of my process.

Replication works fine, I am just looking to capture the logs.

Here is the structure of the distcp action in my workflow.xml:

<action name="distcp-act">
    <distcp xmlns="uri:oozie:distcp-action:0.1">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <configuration>
            <property>
                <name>mapred.job.queue.name</name>
                <value>${queueName}</value>
            </property>
        </configuration>
        <arg>-delete</arg>
        <arg>-overwrite</arg>
	<arg>-log</arg>
	<arg>${logPath}</arg>
        <arg>${srcNode}${srcLocation}</arg>
        <arg>${tgtNode}${tgtLocation}</arg>
    </distcp>
    <ok to="next"/>
    <error to="kill"/>
</action>
3 REPLIES 3

Super Guru

New Contributor

@Kuldeep Kulkarni thank you for your response! The article highlights a great way to piece apart a oozie flow if you are looking for errors in the stack. However, what my question was regarding more was being able to pull the shell logs from distcp actions. Again, I am not having any errors, I simply want to be able to grab the shell logs which show the command line of the action carrying out the distcp. This is simply for logging purposes to go along with the log which captures the rest of my running process. There is a -log option for the Oozie disctp action (as shown above), I am wondering how to properly use that argument to write the action log out to a local log directory.

Super Guru

@Ben Forman - I'm sorry for late reply. Adding my answer for other users who are having same question.

You can add echo statements in your shell scripts and add below tag in your shell action to capture the output which you can get in stdout section of oozie launcher logs.

            <capture-output/>

For more details - Please check https://oozie.apache.org/docs/3.3.0/DG_ShellActionExtension.html

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.