Skip to main content
PCP2
PCP2
Silver ⭐️⭐️
June 17, 2022
Question

Oozie job failed due to below error.

Application application_1655291383854_84179 failed 2 times due to AM Container for appattempt_1655291383854_84179_000002 exited with exitCode: 1 Failing this attempt.Diagnostics: [2022-06-17 00:49:32.617]Exception from container-launch. Container id: container_e1071_1655291383854_84179_02_000001 Exit code: 1 Exception message: Launch container failed Shell output: main : command provided 1 main : run as user is sapproep main : requested yarn user is sapproep Getting exit code file... Creating script paths... Writing pid file... Writing to tmp file /data/data02/yarn/nm/nmPrivate/application_1655291383854_84179/container_e1071_1655291383854_84179_02_000001/container_e1071_1655291383854_84179_02_000001.pid.tmp Writing to cgroup task files... Creating local dirs... Launching container... [2022-06-17 00:49:32.619]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Error un-registering AM client java.lang.NullPointerException at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.unregisterApplicationMaster(AMRMClientImpl.java:613) at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.unregisterApplicationMaster(AMRMClientAsyncImpl.java:220) at org.apache.oozie.action.hadoop.LauncherAM.unregisterWithRM(LauncherAM.java:359) at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:285) at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:144) Exception in thread "main" java.lang.NullPointerException at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.unregisterApplicationMaster(AMRMClientImpl.java:613) at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.unregisterApplicationMaster(AMRMClientAsyncImpl.java:220) at org.apache.oozie.action.hadoop.LauncherAM.unregisterWithRM(LauncherAM.java:359) at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:285) at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:144) [2022-06-17 00:49:32.623]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Error un-registering AM client java.lang.NullPointerException at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.unregisterApplicationMaster(AMRMClientImpl.java:613) at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.unregisterApplicationMaster(AMRMClientAsyncImpl.java:220) at org.apache.oozie.action.hadoop.LauncherAM.unregisterWithRM(LauncherAM.java:359) at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:285) at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:144) Exception in thread "main" java.lang.NullPointerException at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.unregisterApplicationMaster(AMRMClientImpl.java:613) at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.unregisterApplicationMaster(AMRMClientAsyncImpl.java:220) at org.apache.oozie.action.hadoop.LauncherAM.unregisterWithRM(LauncherAM.java:359) at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:285) at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:144) For more detailed output, check the application tracking page: https://itsusralsp07967.genericcompany.com:8090/cluster/app/application_1655291383854_84179 Then click on links to logs of each attempt. . Failing the application.

9 replies

mszurap
Platinum ⭐️
mszurap
Platinum ⭐️
June 17, 2022

Hi @PCP2 , can you clarify which HDP/CDH/CDP version are you using?

Is this a one-off or an intermittent issue or does it always happen? 
Is this affecting only a single job? What kind of an action is Oozie trying to launch?

 

Thanks, Miklos

Translate
PCP2
PCP2Author
Silver ⭐️⭐️
June 17, 2022

Hi @mszurap , we are using CDP version 7.1.7, this is an intermittent issue, encountering twice a week. 

This is a single job(ingestion) where we are processing files(parquet) using shell scripts. Insertion of these files will go through map reduce. 

Translate
mszurap
Platinum ⭐️
mszurap
Platinum ⭐️
June 17, 2022

Can you review the whole logfile? The above NPE may be just a side effect of another failure before.

Translate
Krisssh
Krisssh
Silver ⭐️⭐️
August 15, 2022

@PCP2 Were you able to get any leads on this issue ? 

Translate
DianaTorres
Community Manager
DianaTorres
Community Manager
August 22, 2022

@PCP2 @Krisssh Hello, we cannot work out such issues through a community post, we recommend to open a case to look in detail for each case. Thanks!

Translate
Regards, Diana Torres, Senior Community Moderator
RaghunathJosula
RaghunathJosula
Silver ⭐️
January 22, 2024

Dear All,

I am also facing the same issue with CDP 7.1.7 the oozie job is failing and being killed with the above error.

If any case has been opened please post reply to the same one, this is long pending issue what i am facing with the same log.

Thanks,

Raghunath Josula.

Translate
Raghunath_CDH
Raghunath_CDH
Silver ⭐️⭐️
February 27, 2024

Dear Support what kind of case we should open I don't have any information on this issue please respond.

Translate
    duhizjame
    duhizjame
    Silver ⭐️⭐️
    June 19, 2024

    We are experiencing the same issue on CDP 7.1.7 calling a spark job from Oozie.

    Translate
      Terms & ConditionsAccessibility statement