Oozie Hive Job status changes to KILLED but the underlying job is succes

[#1]

Hello,

I'm running Hive 3.1.3, Hue 4.10.0, Oozie 5.2.1

Through Oozie, I'm running a periodic Hive job which takes data from an HDFS path, and adds it to an existing hive table.

It works fine when I'm running it using Hive shell. However, when I run it using Oozie, the underlying data in Hive gets updated correctly, but the Oozie job fails without giving any meaningful error.

This is a screenshot explaining the situation. The job ran for 1hr 11m, the data in Hive got updated, but Oozie responded with a failed status.

Here are the applicationLogs : 

Parameters:
------------------------
  DATE=2023-04-12
------------------------

Hive command arguments :
             --hiveconf
             hive.log4j.file=/mnt/yarn/usercache/ec2-user/appcache/application_1675167116593_12260/container_1675167116593_12260_01_000001/hive-log4j.properties
             --hiveconf
             hive.exec.log4j.file=/mnt/yarn/usercache/ec2-user/appcache/application_1675167116593_12260/container_1675167116593_12260_01_000001/hive-exec-log4j.properties
             --hiveconf
             hive.log.trace.id=oozie:0005857-230202071432750-oozie-oozi-W
             -f
             dr_update_hive.hql
             --hivevar
             DATE=2023-04-12

Fetching child yarn jobs
tag id : oozie-2621f368aebc0324f3d78a262abee63e
No child applications found
=================================================================

>>> Invoking Hive command line now >>>


<<< Invocation of Hive command completed <<<

No child hadoop job is executed.
java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:412)
	at org.apache.oozie.action.hadoop.LauncherAM.access$400(LauncherAM.java:54)
	at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:225)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
	at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:219)
	at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:155)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
	at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:143)
Caused by: java.lang.SecurityException: Intercepted System.exit(1)
	at org.apache.oozie.action.hadoop.security.LauncherSecurityManager.checkExit(LauncherSecurityManager.java:57)
	at java.lang.Runtime.exit(Runtime.java:107)
	at java.lang.System.exit(System.java:973)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:685)
	at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:310)
	at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:294)
	at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:107)
	at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:69)
	... 16 more
Intercepting System.exit(1)
Failing Oozie Launcher, Main Class [org.apache.oozie.action.hadoop.HiveMain], exit code [1]
Oozie Launcher, uploading action data to HDFS sequence file: hdfs://ip-10-162-8-160.ap-south-1.compute.internal:8020/user/ec2-user/oozie-oozi/0005857-230202071432750-oozie-oozi-W/hiveScript--hive/action-data.seq
Stopping AM
Callback notification attempts left 0
Callback notification trying http://ip-10-162-8-160.ap-south-1.compute.internal:11000/oozie/callback?id=0005857-230202071432750-oozie-oozi-W@hiveScript&status=FAILED
Callback notification to http://ip-10-162-8-160.ap-south-1.compute.internal:11000/oozie/callback?id=0005857-230202071432750-oozie-oozi-W@hiveScript&status=FAILED succeeded
Callback notification succeeded

I need this because the job is correct and I'm getting unnecessary FAILURE alerts.

P.S. : Please tell me if any other info is needed. I'll be happy to provide

[#2]

@yashbansal042  Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our Hive experts @Shmoo and @mszurap  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

[#3]

Thanks @DianaTorres 

[#4 · Accepted solution]

Hello  @yashbansal042 , 

It seems that you are using the "hive" action (Hive1 action) which is using the Hive CLI, that is deprecated and not supported in CDP:

https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/hive-introduction/topics/hive-unsupported.htm... 

Please follow

https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/starting-hive/topics/hive_use_variables.html

to convert your Hive CLI scripts to beeline compatible scripts, and then use the "hive2" action in Oozie.

First, you might want to test whether a simple hive query (like the "select 1" query) works fine through a hive2 action.

Best regards

 Miklos

[#5]

@mszurap Thanks!! It worked. Upgraded to Hive2 action and things worked.

However, had to add this property in hive hql : 

SET hive.stats.autogather=false;

Guess, the flow was failing due to some issues with stats gathering.

We can mark this as resolved.

[#6]

Glad to hear that. 🙂

The "hive.stats.autogather" should be unrelated to this, it just controls whether Hive should gather statistics at the end of INSERT statements, disabling it just speeds up the insert query somewhat. If you see something weird or not working with that enabled, feel free to open a support case on our Support Portal.

See this and all the hive configuration descriptions in the HiveConf.java source code.

Cheers,

 Miklos