Skip to main content
yashbansal042
Silver ⭐️⭐️
Apr 13, 2023 2:21 PM UTC
Solved

Oozie Hive Job status changes to KILLED but the underlying job is succes

Hello,

I'm running Hive 3.1.3, Hue 4.10.0, Oozie 5.2.1

 

Through Oozie, I'm running a periodic Hive job which takes data from an HDFS path, and adds it to an existing hive table.

It works fine when I'm running it using Hive shell. However, when I run it using Oozie, the underlying data in Hive gets updated correctly, but the Oozie job fails without giving any meaningful error.

 

This is a screenshot explaining the situation. The job ran for 1hr 11m, the data in Hive got updated, but Oozie responded with a failed status.

Screenshot 2023-04-13 at 7.47.32 PM.png

Here are the applicationLogs : 

 

Parameters:
------------------------
DATE=2023-04-12
------------------------

Hive command arguments :
--hiveconf
hive.log4j.file=/mnt/yarn/usercache/ec2-user/appcache/application_1675167116593_12260/container_1675167116593_12260_01_000001/hive-log4j.properties
--hiveconf
hive.exec.log4j.file=/mnt/yarn/usercache/ec2-user/appcache/application_1675167116593_12260/container_1675167116593_12260_01_000001/hive-exec-log4j.properties
--hiveconf
hive.log.trace.id=oozie:0005857-230202071432750-oozie-oozi-W
-f
dr_update_hive.hql
--hivevar
DATE=2023-04-12

Fetching child yarn jobs
tag id : oozie-2621f368aebc0324f3d78a262abee63e
No child applications found
=================================================================

>>> Invoking Hive command line now >>>


<<< Invocation of Hive command completed <<<

No child hadoop job is executed.
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:412)
at org.apache.oozie.action.hadoop.LauncherAM.access$400(LauncherAM.java:54)
at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:225)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:219)
at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:155)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:143)
Caused by: java.lang.SecurityException: Intercepted System.exit(1)
at org.apache.oozie.action.hadoop.security.LauncherSecurityManager.checkExit(LauncherSecurityManager.java:57)
at java.lang.Runtime.exit(Runtime.java:107)
at java.lang.System.exit(System.java:973)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:685)
at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:310)
at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:294)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:107)
at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:69)
... 16 more
Intercepting System.exit(1)
Failing Oozie Launcher, Main Class [org.apache.oozie.action.hadoop.HiveMain], exit code [1]
Oozie Launcher, uploading action data to HDFS sequence file: hdfs://ip-10-162-8-160.ap-south-1.compute.internal:8020/user/ec2-user/oozie-oozi/0005857-230202071432750-oozie-oozi-W/hiveScript--hive/action-data.seq
Stopping AM
Callback notification attempts left 0
Callback notification trying http://ip-10-162-8-160.ap-south-1.compute.internal:11000/oozie/callback?id=0005857-230202071432750-oozie-oozi-W@hiveScript&status=FAILED
Callback notification to http://ip-10-162-8-160.ap-south-1.compute.internal:11000/oozie/callback?id=0005857-230202071432750-oozie-oozi-W@hiveScript&status=FAILED succeeded
Callback notification succeeded

 

 

I need this because the job is correct and I'm getting unnecessary FAILURE alerts.

P.S. : Please tell me if any other info is needed. I'll be happy to provide

mszurap
mszurapAnswer
Platinum ⭐️
Apr 17, 2023 9:41 AM UTC

Hello  @yashbansal042 , 

It seems that you are using the "hive" action (Hive1 action) which is using the Hive CLI, that is deprecated and not supported in CDP:

https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/hive-introduction/topics/hive-unsupported.html 

Please follow

https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/starting-hive/topics/hive_use_variables.html

to convert your Hive CLI scripts to beeline compatible scripts, and then use the "hive2" action in Oozie.

First, you might want to test whether a simple hive query (like the "select 1" query) works fine through a hive2 action.

Best regards

 Miklos

5 replies

DianaTorres
Community Manager
Community Manager
Apr 14, 2023 5:28 PM UTC

@yashbansal042  Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our Hive experts @Shmoo and @mszurap  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Translate
Regards, Diana Torres, Senior Community Moderator
yashbansal042
Silver ⭐️⭐️
Apr 17, 2023 4:16 AM UTC

Thanks @DianaTorres 

 

Translate
mszurap
Platinum ⭐️
mszurapAnswer
Platinum ⭐️
Apr 17, 2023 9:41 AM UTC

Hello  @yashbansal042 , 

It seems that you are using the "hive" action (Hive1 action) which is using the Hive CLI, that is deprecated and not supported in CDP:

https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/hive-introduction/topics/hive-unsupported.html 

Please follow

https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/starting-hive/topics/hive_use_variables.html

to convert your Hive CLI scripts to beeline compatible scripts, and then use the "hive2" action in Oozie.

First, you might want to test whether a simple hive query (like the "select 1" query) works fine through a hive2 action.

Best regards

 Miklos

Translate
yashbansal042
Silver ⭐️⭐️
Apr 19, 2023 3:55 AM UTC

@mszurap Thanks!! It worked. Upgraded to Hive2 action and things worked.

However, had to add this property in hive hql : 

SET hive.stats.autogather=false;

Guess, the flow was failing due to some issues with stats gathering.

 

We can mark this as resolved.

Translate
mszurap
Platinum ⭐️
Platinum ⭐️
Apr 19, 2023 7:22 AM UTC

Glad to hear that. 🙂

The "hive.stats.autogather" should be unrelated to this, it just controls whether Hive should gather statistics at the end of INSERT statements, disabling it just speeds up the insert query somewhat. If you see something weird or not working with that enabled, feel free to open a support case on our Support Portal.

See this and all the hive configuration descriptions in the HiveConf.java source code.

Cheers,

 Miklos

Translate