Reply
Expert Contributor
Posts: 88
Registered: ‎01-31-2015

Oozie-hive jobs hanging for longtime with "heartbeat" after upgrade to CDH 5.11.1

We have upgraded our CDH Cluster from version 5.8.3 to CDH 5.11.1 We are currently seeing our oozie-hive query jobs hanging forever which never used to happen in the previous CDH versions. Same code used in previous version never caused this issue. Is there anything changed in the new version, do we have to set some extra configurations, Please help us as early as possible to resolve this issue.

 

Sample xml:

 

<action name="abc-incr">

<hive2 xmlns="uri:oozie:hive2-action:0.2">

<job-tracker>${jobTracker}</job-tracker>

<name-node>${nameNode}</name-node>

<job-xml>${hiveConfDir}/hive-site.xml</job-xml>

<jdbc-url>${beeline_jdbc_url}</jdbc-url>

<script>path/hql/abc.hql</script>

<param>databasedestination=${abc}</param>

<param>id=${Id}</param>

<param>databasesource=${abc1}</param>

<argument>-wpf</argument>

<file>${hiveConfDir}/hive-site.xml#hive-site.xml</file>

<file>${nameNode}${impala_udfs}/pf#pf</file>

</hive2>

<ok to="impala_refresh" />

<error to="killEmail" />

</action>

Posts: 11
Topics: 0
Kudos: 1
Solutions: 0
Registered: ‎06-14-2017

Re: Oozie-hive jobs hanging for longtime with "heartbeat" after upgrade to CDH 5.11.1

Hi Jais,

 

What is the state of the Yarn application launched for processing the Hive query ?

Is it : accepted, running, success, failed, killed ?

 

If it's in the state "accepted" then this is a Yarn configuration issue.

For the other states, you will need to check the logs in order to understand what is happening :

- of each tasks

- of the HiveServer2 server used

 

 

Expert Contributor
Posts: 88
Registered: ‎01-31-2015

Re: Oozie-hive jobs hanging for longtime with "heartbeat" after upgrade to CDH 5.11.1

Yes, its in accepted stated, waititng to execute the query with following status "hear beat ........."

Expert Contributor
Posts: 88
Registered: ‎01-31-2015

Re: Oozie-hive jobs hanging for longtime with "heartbeat" after upgrade to CDH 5.11.1

We are still seeing this issue, to make it clear the Hive queries run fine for couple of days in normal duration through oozie workflows, after then they start slowing down, once we restart hive service, they come back to normal run, again run for couple of days in normal duration then same thing repeats of slowing down, but all other services like java process , impala queries have no impact, they run at same way as they were before.

 

We never came across this issue in our previous version of CDH, the same set of queries used to run fine with same configurations without any problems on CDH 5.8.3.

 

Please need some one quickly to help us out in resolving this issue.

Champion
Posts: 563
Registered: ‎05-16-2016

Re: Oozie-hive jobs hanging for longtime with "heartbeat" after upgrade to CDH 5.11.1

@Jais 

what os  version and kernel are you ruining ? 

when did you upgrade your CDH ? 

did you had a chance to check the HS2 logs 

also please give us the Resourcemanager logs to dig in more . 

Announcements