Could not run Oozie workflow with distcp-action

[#1]

Hi,

I am trying to run Oozie workflow job with distcp-action but getting below error:

2015-12-30 10:19:36,004  INFO CallbackServlet:520 - SERVER[centos10-82.letv.cn] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000013-151228182829447-oozie-oozi-W] ACTION[0000013-151228182829447-oozie-oozi-W@distcp-node] callback for action [0000013-151228182829447-oozie-oozi-W@distcp-node]
2015-12-30 10:19:36,138  INFO DistcpActionExecutor:520 - SERVER[centos10-82.letv.cn] USER[oozie] GROUP[-] TOKEN[] APP[distcp-WF] JOB[0000013-151228182829447-oozie-oozi-W] ACTION[0000013-151228182829447-oozie-oozi-W@distcp-node] action completed, external ID [job_1450278216920_52402]
2015-12-30 10:19:36,152  WARN DistcpActionExecutor:523 - SERVER[centos10-82.letv.cn] USER[oozie] GROUP[-] TOKEN[] APP[distcp-WF] JOB[0000013-151228182829447-oozie-oozi-W] ACTION[0000013-151228182829447-oozie-oozi-W@distcp-node] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.DistcpMain], exception invoking main(), java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.DistcpMain not found
2015-12-30 10:19:36,152  WARN DistcpActionExecutor:523 - SERVER[centos10-82.letv.cn] USER[oozie] GROUP[-] TOKEN[] APP[distcp-WF] JOB[0000013-151228182829447-oozie-oozi-W] ACTION[0000013-151228182829447-oozie-oozi-W@distcp-node] Launcher exception: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.DistcpMain not found
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.DistcpMain not found
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)
        at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:234)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.DistcpMain not found
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
        ... 9 more
2015-12-30 10:19:36,179  INFO ActionEndXCommand:520 - SERVER[centos10-82.letv.cn] USER[oozie] GROUP[-] TOKEN[] APP[distcp-WF] JOB[0000013-151228182829447-oozie-oozi-W] ACTION[0000013-151228182829447-oozie-oozi-W@distcp-node] ERROR is considered as FAILED for SLA
2015-12-30 10:19:36,196  INFO ActionStartXCommand:520 - SERVER[centos10-82.letv.cn] USER[oozie] GROUP[-] TOKEN[] APP[distcp-WF] JOB[0000013-151228182829447-oozie-oozi-W] ACTION[0000013-151228182829447-oozie-oozi-W@fail] Start action [0000013-151228182829447-oozie-oozi-W@fail] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]

and the workflow.xml file content as below:

<workflow-app xmlns="uri:oozie:workflow:0.4" name="distcp-WF">
    <start to="distcp-node"/>
    <action name="distcp-node">
        <distcp xmlns="uri:oozie:distcp-action:0.1">
            <job-tracker>${jobTracker}</job-tracker>
            <prepare>
<delete path="${to}"/>
</prepare>            <configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
               <property>
                   <name>oozie.use.system.libpath</name>
                   <value>true</value>
               </property>            </configuration>
            <arg>${from}</arg>
            <arg>${to}</arg>
        </distcp>
        <ok to="end"/>
        <error to="fail"/>
    </action>
    <kill name="fail">
        <message>DistCP failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

then, i checked the SharedLib in these path:

• ls /usr/hdp/2.3.2.0-2950/oozie/share/lib/distcp

• hdfs oozie ShareLib:

• 

please help me, Thanks!

[#2 · Accepted solution]

@Hefei Li

It appears that you have the jars at the right place but it could be a permissions issue. I see in your screen snapshots that the jar file have the 644 permissions but how about the directory containing the jar files? The directories are recommended to have 755.

This issue could occur due to incorrect value of umask (recommended value 0022)

Fix: Try setting the permissions of all directories in path to 755 and try again.

Let us know how it goes.

[#3]

i changed all jars permissions, and the result like below:

then, i run the job again, but it still not works ok, and the error log as below:

[#4]

@Hefei Li

Can you set the directory permissions to 755? (all the directories ,including parents, in the path that contains the jar files)

[#5]

@bsaini

yes, i am sure.

but i have a puzzle. try it like above, is need to do on each machine in the Hadoop cluster, or just do it on Oozie server machine.

[#6]

@bsaini

Hi bsaini. I finally sloved it.

Like you said, is a permissions issue. All the jar files which installed in HDFS must be set to 755.

My HDP version is 2.3.x, and the default Oozie Share Library is hdfs://{my_host}:8020//user/oozie/share/lib/lib_20151119202305/oozie.

Thank you for your guidance.

[#7]

@Hefei Li

Great! Can you accept the answer then so we can close this question and others having similar issue get benefited?