​Hive queries blocked - Tez always shows running

[#1]

I have a Hive query (coming from Nifi), it is an insert operation, and Tez (view from Ambari) has reported it running for >48 hours. This was preventing other operations from occuring, so I wanted to kill the query so the backlog could be processed.

I killed the associated Yarn Application. The Tez view from Ambari still shows the status as Running. Clicking through to the Application ID shows the status as Killed and Final Status as Undefined. Clicking through to the DAG shows the status as Killed, and the only listed Vertex is also Killed. Total Tasks for the DAG is 1.

However, despite all this the query is still marked as running in TEZ and other queries are blocked behind it. Even non-modifying queries like a simple SELECT COUNT(*) are unable to run. This has persisted across Yarn, Hive, Tez, and even entire machine restarts.

Is there some way to kill off the operations that Tez is showing? Killing the yarn application doesn't seem to be sufficient.

Versions:
YARN: 2.7.3
Tez: 0.7.0
Hive: 1.2.1000

[#2]

@Nick Price

Which version ?

[#3]

@Vinod Bonthu
Sorry, updated post to reflect versions

[#4]

I'm experiencing this same issue btw. I'm wondering if we're running out of resources for tez. I have a support case open with Hortonworks and will let you know what comes of it. We usually have to restart Hive in order to for more Tez/Hive jobs to run.

[#5]

In our case, restarting Hive (or the entire stack for that matter) didn't seem to do anything. After playing with some resource allocation, we were able to get some jobs to go through, but the ones that were killed still remain.

I'm starting to wonder if it's an issue with Tez not recognizing that jobs are being killed.

[#6]

Came here to find how to kill queries that show as running in Tez. I am facing the same issue as documented above..Tez shows 4 queries running without ApplicationID and DAGID's.

I have restarted the whole suite, but still shows running.

Versions:
YARN: 2.6.3.0-235
Tez-client - 2.6.3.0-235

Hadoop 2.7.3.2.6.3.0-235

[#7]

I'm having the same problem, I killed some jobs, but tez UI saus some of them are still running; no App ID or DAG ID. Query was launched from Hive

HDP 2.6.0.3-8

YARN 2.7.3

Tez 0.7.0

[#8]

We are facing the same issue. Restarting the cluster doesn't seem to help either. Has anyone been able to figure out a way to clear those stale running queries?

[#9]

In our case it was a resource allocation issue. We had our queues not set up optimally and we also changed some memory configurations around to allow the initial query process to run properly. Basically, the default queue didn't have any resources and we noticed that when a query first is submitted, there's a process that runs in the default queue before submitting the job to the queue specified.

[#10]

We are facing the same issue. is it a Bug?

[#11]

I am also facing the same issue, did you find a resolution?

[#12]

I have the same issue

[#13]

I have the same problem.

A little of help here would be nice

[#14]

This board does not seem to be very helpful to anyone experiencing issues with the Hortonworks tutorials. On a few occasions I have received meaningful assistance from others, but most of the issues raised receive only a cursory response if any.

[#15]

This I believe, is in correlation to the following bug Link which was corrected in specific versions of Tez.

If you are running Interactive query HDInsight in Azure, they are still using Tez .7.0 (not fixed).

I did however, find this blog post that explains how to at least stop the queries from running. As far as I can tell, Tez will still show them as running, but it does free up your resources again.

Edit: the follow up link is probably helpful 🙂 Fix