Reply
Highlighted
Explorer
Posts: 14
Registered: ‎08-12-2014

Hive Metastore Takes Two Hours to Start

We have thousands of tables and hundreds of partitions for each of those tables.

After enabling Sentry and HDFS ACL sync, it now take 2 full hours before Hive is usable again after restarting.

 

Metastore logs are filled with log entries that look like this:

Jul 20, 3:00:02.313 PM INFO org.apache.hadoop.hive.metastore.HiveMetaStore 9: get_partitions_by_names : db=**** tbl=*****
Jul 20, 3:00:02.313 PM INFO org.apache.hadoop.hive.metastore.HiveMetaStore.audit ugi=hive ip=unknown-ip-addr cmd=get_partitions_by_names : db=**** tbl=****

 

It looks like it is calling the get_partitions_by_names 10 times per table.

 

Is there any way to get the HMS to load faster?

 

/* Joey */

Announcements