question Re: Oozie Hive Job status changes to KILLED but the underlying job is succes in Support Questions

Oozie Hive Job status changes to KILLED but the underlying job is succes

yashbansal042 — Thu, 13 Apr 2023 14:21:32 GMT

Hello,

I'm running Hive 3.1.3, Hue 4.10.0, Oozie 5.2.1

Through Oozie, I'm running a periodic Hive job which takes data from an HDFS path, and adds it to an existing hive table.

It works fine when I'm running it using Hive shell. However, when I run it using Oozie, the underlying data in Hive gets updated correctly, but the Oozie job fails without giving any meaningful error.

This is a screenshot explaining the situation. The job ran for 1hr 11m, the data in Hive got updated, but Oozie responded with a failed status.

Here are the applicationLogs :

Parameters: ------------------------ DATE=2023-04-12 ------------------------ Hive command arguments : --hiveconf hive.log4j.file=/mnt/yarn/usercache/ec2-user/appcache/application_1675167116593_12260/container_1675167116593_12260_01_000001/hive-log4j.properties --hiveconf hive.exec.log4j.file=/mnt/yarn/usercache/ec2-user/appcache/application_1675167116593_12260/container_1675167116593_12260_01_000001/hive-exec-log4j.properties --hiveconf hive.log.trace.id=oozie:0005857-230202071432750-oozie-oozi-W -f dr_update_hive.hql --hivevar DATE=2023-04-12 Fetching child yarn jobs tag id : oozie-2621f368aebc0324f3d78a262abee63e No child applications found ================================================================= >>> Invoking Hive command line now >>> <<< Invocation of Hive command completed <<< No child hadoop job is executed. java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:412) at org.apache.oozie.action.hadoop.LauncherAM.access$400(LauncherAM.java:54) at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:225) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:219) at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:155) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:143) Caused by: java.lang.SecurityException: Intercepted System.exit(1) at org.apache.oozie.action.hadoop.security.LauncherSecurityManager.checkExit(LauncherSecurityManager.java:57) at java.lang.Runtime.exit(Runtime.java:107) at java.lang.System.exit(System.java:973) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:685) at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:310) at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:294) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:107) at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:69) ... 16 more Intercepting System.exit(1) Failing Oozie Launcher, Main Class [org.apache.oozie.action.hadoop.HiveMain], exit code [1] Oozie Launcher, uploading action data to HDFS sequence file: hdfs://ip-10-162-8-160.ap-south-1.compute.internal:8020/user/ec2-user/oozie-oozi/0005857-230202071432750-oozie-oozi-W/hiveScript--hive/action-data.seq Stopping AM Callback notification attempts left 0 Callback notification trying http://ip-10-162-8-160.ap-south-1.compute.internal:11000/oozie/callback?id=0005857-230202071432750-oozie-oozi-W@hiveScript&status=FAILED Callback notification to http://ip-10-162-8-160.ap-south-1.compute.internal:11000/oozie/callback?id=0005857-230202071432750-oozie-oozi-W@hiveScript&status=FAILED succeeded Callback notification succeeded

I need this because the job is correct and I'm getting unnecessary FAILURE alerts.

P.S. : Please tell me if any other info is needed. I'll be happy to provide

Re: Oozie Hive Job status changes to KILLED but the underlying job is succes

DianaTorres — Fri, 14 Apr 2023 17:28:25 GMT

@yashbansal042 Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our Hive experts @Shmoo and @mszurap who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Re: Oozie Hive Job status changes to KILLED but the underlying job is succes

yashbansal042 — Mon, 17 Apr 2023 04:16:28 GMT

Thanks @DianaTorres

Re: Oozie Hive Job status changes to KILLED but the underlying job is succes

mszurap — Mon, 17 Apr 2023 09:41:16 GMT

Hello @yashbansal042 ,

It seems that you are using the "hive" action (Hive1 action) which is using the Hive CLI, that is deprecated and not supported in CDP:

https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/hive-introduction/topics/hive-unsupported.html

Please follow

https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/starting-hive/topics/hive_use_variables.html

to convert your Hive CLI scripts to beeline compatible scripts, and then use the "hive2" action in Oozie.

First, you might want to test whether a simple hive query (like the "select 1" query) works fine through a hive2 action.

Best regards

Miklos

Re: Oozie Hive Job status changes to KILLED but the underlying job is succes

yashbansal042 — Wed, 19 Apr 2023 03:55:00 GMT

@mszurap Thanks!! It worked. Upgraded to Hive2 action and things worked.

However, had to add this property in hive hql :

SET hive.stats.autogather=false;

Guess, the flow was failing due to some issues with stats gathering.

We can mark this as resolved.

Re: Oozie Hive Job status changes to KILLED but the underlying job is succes

mszurap — Wed, 19 Apr 2023 07:22:26 GMT

Glad to hear that. 🙂

The "hive.stats.autogather" should be unrelated to this, it just controls whether Hive should gather statistics at the end of INSERT statements, disabling it just speeds up the insert query somewhat. If you see something weird or not working with that enabled, feel free to open a support case on our Support Portal.

See this and all the hive configuration descriptions in the HiveConf.java source code.

Cheers,

Miklos