Oozie job failed due to below error.

[#1]

Application application_1655291383854_84179 failed 2 times due to AM Container for appattempt_1655291383854_84179_000002 exited with exitCode: 1 Failing this attempt.Diagnostics: [2022-06-17 00:49:32.617]Exception from container-launch. Container id: container_e1071_1655291383854_84179_02_000001 Exit code: 1 Exception message: Launch container failed Shell output: main : command provided 1 main : run as user is sapproep main : requested yarn user is sapproep Getting exit code file... Creating script paths... Writing pid file... Writing to tmp file /data/data02/yarn/nm/nmPrivate/application_1655291383854_84179/container_e1071_1655291383854_84179_0... Writing to cgroup task files... Creating local dirs... Launching container... [2022-06-17 00:49:32.619]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Error un-registering AM client java.lang.NullPointerException at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.unregisterApplicationMaster(AMRMClientImpl.java:613) at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.unregisterApplicationMaster(AMRMClientAsyncImpl.java:220) at org.apache.oozie.action.hadoop.LauncherAM.unregisterWithRM(LauncherAM.java:359) at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:285) at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:144) Exception in thread "main" java.lang.NullPointerException at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.unregisterApplicationMaster(AMRMClientImpl.java:613) at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.unregisterApplicationMaster(AMRMClientAsyncImpl.java:220) at org.apache.oozie.action.hadoop.LauncherAM.unregisterWithRM(LauncherAM.java:359) at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:285) at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:144) [2022-06-17 00:49:32.623]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Error un-registering AM client java.lang.NullPointerException at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.unregisterApplicationMaster(AMRMClientImpl.java:613) at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.unregisterApplicationMaster(AMRMClientAsyncImpl.java:220) at org.apache.oozie.action.hadoop.LauncherAM.unregisterWithRM(LauncherAM.java:359) at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:285) at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:144) Exception in thread "main" java.lang.NullPointerException at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.unregisterApplicationMaster(AMRMClientImpl.java:613) at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.unregisterApplicationMaster(AMRMClientAsyncImpl.java:220) at org.apache.oozie.action.hadoop.LauncherAM.unregisterWithRM(LauncherAM.java:359) at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:285) at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:144) For more detailed output, check the application tracking page: https://itsusralsp07967.genericcompany.com:8090/cluster/app/application_1655291383854_84179 Then click on links to logs of each attempt. . Failing the application.

[#2]

Hi @PCP2 , can you clarify which HDP/CDH/CDP version are you using?

Is this a one-off or an intermittent issue or does it always happen? 
Is this affecting only a single job? What kind of an action is Oozie trying to launch?

Thanks, Miklos

[#3]

Hi @mszurap , we are using CDP version 7.1.7, this is an intermittent issue, encountering twice a week. 

This is a single job(ingestion) where we are processing files(parquet) using shell scripts. Insertion of these files will go through map reduce. 

[#4]

Can you review the whole logfile? The above NPE may be just a side effect of another failure before.

[#5]

We dont have much information in logs. we refer stderr logs...PFA logs for the failed instance.

[#6]

@PCP2 Were you able to get any leads on this issue ? 

[#7]

@PCP2 @Krisssh Hello, we cannot work out such issues through a community post, we recommend to open a case to look in detail for each case. Thanks!

[#8]

Dear All,

I am also facing the same issue with CDP 7.1.7 the oozie job is failing and being killed with the above error.

If any case has been opened please post reply to the same one, this is long pending issue what i am facing with the same log.

Thanks,

Raghunath Josula.

[#9]

Dear Support what kind of case we should open I don't have any information on this issue please respond.

[#10]

We are experiencing the same issue on CDP 7.1.7 calling a spark job from Oozie.