Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Catch complex Hive queries before submission

Highlighted

Catch complex Hive queries before submission

Super Collaborator

Hi,

I am thinking of a way to assess Hive queries before they are submitted based on some anticipated process time. So assuming statistics are gathered regularly, cbo is enabled etc. what would be a good way to summarize all the info 'explain select.... ' spits out into 1 KPI.

The eventual aim is to assign high costs queries into a separate queue (mapreduce) and low cost to Tez.

-

1 REPLY 1

Re: Catch complex Hive queries before submission

Rising Star

You can look at the Hive hooks. I.e. you can write Java classes which are called before/after/on failure according to how you configure the proper variables in Hive configuration. Here you can find some slides about them: http://www.slideshare.net/julingks/apache-hive-hooksminwookim130813.

Here you can also find an article with an example about how to change the queue a query is submitted to: https://community.hortonworks.com/articles/24009/map-hive-jobs-to-yarn-queues.html.

Don't have an account?
Coming from Hortonworks? Activate your account here