Created on 12-30-2015 03:17 AM - edited 08-19-2019 05:22 AM
Hi,
I am trying to run Oozie workflow job with distcp-action but getting below error:
2015-12-30 10:19:36,004 INFO CallbackServlet:520 - SERVER[centos10-82.letv.cn] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000013-151228182829447-oozie-oozi-W] ACTION[0000013-151228182829447-oozie-oozi-W@distcp-node] callback for action [0000013-151228182829447-oozie-oozi-W@distcp-node] 2015-12-30 10:19:36,138 INFO DistcpActionExecutor:520 - SERVER[centos10-82.letv.cn] USER[oozie] GROUP[-] TOKEN[] APP[distcp-WF] JOB[0000013-151228182829447-oozie-oozi-W] ACTION[0000013-151228182829447-oozie-oozi-W@distcp-node] action completed, external ID [job_1450278216920_52402] 2015-12-30 10:19:36,152 WARN DistcpActionExecutor:523 - SERVER[centos10-82.letv.cn] USER[oozie] GROUP[-] TOKEN[] APP[distcp-WF] JOB[0000013-151228182829447-oozie-oozi-W] ACTION[0000013-151228182829447-oozie-oozi-W@distcp-node] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.DistcpMain], exception invoking main(), java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.DistcpMain not found 2015-12-30 10:19:36,152 WARN DistcpActionExecutor:523 - SERVER[centos10-82.letv.cn] USER[oozie] GROUP[-] TOKEN[] APP[distcp-WF] JOB[0000013-151228182829447-oozie-oozi-W] ACTION[0000013-151228182829447-oozie-oozi-W@distcp-node] Launcher exception: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.DistcpMain not found java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.DistcpMain not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:234) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.DistcpMain not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193) ... 9 more 2015-12-30 10:19:36,179 INFO ActionEndXCommand:520 - SERVER[centos10-82.letv.cn] USER[oozie] GROUP[-] TOKEN[] APP[distcp-WF] JOB[0000013-151228182829447-oozie-oozi-W] ACTION[0000013-151228182829447-oozie-oozi-W@distcp-node] ERROR is considered as FAILED for SLA 2015-12-30 10:19:36,196 INFO ActionStartXCommand:520 - SERVER[centos10-82.letv.cn] USER[oozie] GROUP[-] TOKEN[] APP[distcp-WF] JOB[0000013-151228182829447-oozie-oozi-W] ACTION[0000013-151228182829447-oozie-oozi-W@fail] Start action [0000013-151228182829447-oozie-oozi-W@fail] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
and the workflow.xml file content as below:
<workflow-app xmlns="uri:oozie:workflow:0.4" name="distcp-WF"> <start to="distcp-node"/> <action name="distcp-node"> <distcp xmlns="uri:oozie:distcp-action:0.1"> <job-tracker>${jobTracker}</job-tracker> <prepare> <delete path="${to}"/> </prepare> <configuration> <property> <name>mapred.job.queue.name</name> <value>${queueName}</value> </property> <property> <name>oozie.use.system.libpath</name> <value>true</value> </property> </configuration> <arg>${from}</arg> <arg>${to}</arg> </distcp> <ok to="end"/> <error to="fail"/> </action> <kill name="fail"> <message>DistCP failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <end name="end"/> </workflow-app>
then, i checked the SharedLib in these path:
please help me, Thanks!
Created 12-30-2015 03:32 AM
It appears that you have the jars at the right place but it could be a permissions issue. I see in your screen snapshots that the jar file have the 644 permissions but how about the directory containing the jar files? The directories are recommended to have 755.
This issue could occur due to incorrect value of umask (recommended value 0022)
Fix: Try setting the permissions of all directories in path to 755 and try again.
Let us know how it goes.
Created 12-30-2015 03:32 AM
It appears that you have the jars at the right place but it could be a permissions issue. I see in your screen snapshots that the jar file have the 644 permissions but how about the directory containing the jar files? The directories are recommended to have 755.
This issue could occur due to incorrect value of umask (recommended value 0022)
Fix: Try setting the permissions of all directories in path to 755 and try again.
Let us know how it goes.
Created on 12-30-2015 04:07 AM - edited 08-19-2019 05:22 AM
i changed all jars permissions, and the result like below:
then, i run the job again, but it still not works ok, and the error log as below:
Created 12-30-2015 05:50 AM
Can you set the directory permissions to 755? (all the directories ,including parents, in the path that contains the jar files)
Created 12-30-2015 06:12 AM
yes, i am sure.
but i have a puzzle. try it like above, is need to do on each machine in the Hadoop cluster, or just do it on Oozie server machine.
Created 12-31-2015 10:38 AM
Hi bsaini. I finally sloved it.
Like you said, is a permissions issue. All the jar files which installed in HDFS must be set to 755.
My HDP version is 2.3.x, and the default Oozie Share Library is hdfs://{my_host}:8020//user/oozie/share/lib/lib_20151119202305/oozie.
Thank you for your guidance.
Created 12-31-2015 02:12 PM
Great! Can you accept the answer then so we can close this question and others having similar issue get benefited?