Created 09-06-2017 02:32 PM
I have a Hive query (coming from Nifi), it is an insert operation, and Tez (view from Ambari) has reported it running for >48 hours. This was preventing other operations from occuring, so I wanted to kill the query so the backlog could be processed.
I killed the associated Yarn Application. The Tez view from Ambari still shows the status as Running. Clicking through to the Application ID shows the status as Killed and Final Status as Undefined. Clicking through to the DAG shows the status as Killed, and the only listed Vertex is also Killed. Total Tasks for the DAG is 1.
However, despite all this the query is still marked as running in TEZ and other queries are blocked behind it. Even non-modifying queries like a simple SELECT COUNT(*) are unable to run. This has persisted across Yarn, Hive, Tez, and even entire machine restarts.
Is there some way to kill off the operations that Tez is showing? Killing the yarn application doesn't seem to be sufficient.
Created 09-06-2017 03:25 PM
Created 09-06-2017 03:38 PM
Created 10-19-2017 02:47 AM
I'm experiencing this same issue btw. I'm wondering if we're running out of resources for tez. I have a support case open with Hortonworks and will let you know what comes of it. We usually have to restart Hive in order to for more Tez/Hive jobs to run.
Created 10-19-2017 12:23 PM
In our case, restarting Hive (or the entire stack for that matter) didn't seem to do anything. After playing with some resource allocation, we were able to get some jobs to go through, but the ones that were killed still remain.
I'm starting to wonder if it's an issue with Tez not recognizing that jobs are being killed.
Created 01-09-2018 11:37 PM
Came here to find how to kill queries that show as running in Tez. I am facing the same issue as documented above..Tez shows 4 queries running without ApplicationID and DAGID's.
I have restarted the whole suite, but still shows running.
Tez-client - 188.8.131.52-235
Created on 02-07-2018 06:43 AM - edited 08-18-2019 02:07 AM
I'm having the same problem, I killed some jobs, but tez UI saus some of them are still running; no App ID or DAG ID. Query was launched from Hive
Created 01-29-2018 04:22 PM
We are facing the same issue. Restarting the cluster doesn't seem to help either. Has anyone been able to figure out a way to clear those stale running queries?
Created 01-29-2018 04:29 PM
In our case it was a resource allocation issue. We had our queues not set up optimally and we also changed some memory configurations around to allow the initial query process to run properly. Basically, the default queue didn't have any resources and we noticed that when a query first is submitted, there's a process that runs in the default queue before submitting the job to the queue specified.
Created 03-15-2018 09:24 AM
We are facing the same issue. is it a Bug?
Created 04-14-2018 02:54 PM
I am also facing the same issue, did you find a resolution?
Created 04-20-2018 03:10 PM
I have the same issue
Created 05-15-2018 07:54 AM
I have the same problem.
A little of help here would be nice
Created 05-21-2018 02:11 PM
This board does not seem to be very helpful to anyone experiencing issues with the Hortonworks tutorials. On a few occasions I have received meaningful assistance from others, but most of the issues raised receive only a cursory response if any.
Created 07-19-2018 10:22 PM
This I believe, is in correlation to the following bug Link which was corrected in specific versions of Tez.
If you are running Interactive query HDInsight in Azure, they are still using Tez .7.0 (not fixed).
I did however, find this blog post that explains how to at least stop the queries from running. As far as I can tell, Tez will still show them as running, but it does free up your resources again.
Edit: the follow up link is probably helpful 🙂 Fix