Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Gather information about hive table execution?

Gather information about hive table execution?

Rising Star

HI,

Our team working on Hadoop framework/technologies with Hive, Ambari, Ranger and planning to create a dashboard, which provides following information :

  • Execution time of a Hive query.
  • Size of data generated by the created table.
  • Frequency of using each hive schema, table and each column of respective table.
  • User/ Application name or ID firing any query.
  • Resources usage of each application /User.

For now, we are trying to use HiveMetastore and Ranger audit logs to access the above mentioned information.

Is there any other better way to fetch the information above ?

Kindly let me know, if I need to provide any more information.

1 REPLY 1
Highlighted

Re: Gather information about hive table execution?

Cloudera Employee

You might want to take a look at using the Tez UI, which for HDP 2.6 should have execution time, user, db/table usage, user, and resource usage. This is more focused on queries which run with the Tez execution engine. This makes use of information sent to the Yarn Timeline Server through both Hive (using the ATSHook) and Tez. You could also look into querying the Timeline Server directly for this data as well. There have been a number of updates to the information sent to ATS through Hive/Tez recently, to go into HDP 2.6