question CBO for Hive over hbase in Archives of Support Questions (Read Only)

CBO for Hive over hbase

michelsumbul — Tue, 21 Jun 2016 16:59:04 GMT

Hi,

I have Hive running over HBase table. Is it interesting in this case to analyze the table for the CBO? Or is it only interesting whe nyou have hive with orc file?

Thanks in advance,

Michel

Re: CBO for Hive over hbase

nmaillard1 — Tue, 21 Jun 2016 17:03:03 GMT

Hello Michel

Right now the Hive plan calculation does not reach out to get Hbases stats so currently no added benefit from the Hbase stats. This being said these are questions that are being worked on in different initiatives, so this will likely change in the future.

Re: CBO for Hive over hbase

ssubhas — Tue, 21 Jun 2016 17:05:06 GMT

@Michel Sumbul

CBO is mainly for optimization decisions which reduces the cost of query execution and is independent of storage formats like ORC. Below is some of decisions based on CBO:

How to order Join
What algorithm to use for a given Join
Should the intermediate result be persisted or should it be recomputed on operator failure.
The degree of parallelism at any operator (specifically number of reducers to use).
Semi Join selection

For details, please refer to below link:

https://cwiki.apache.org/confluence/display/Hive/Cost-based+optimization+in+Hive

Thanks and Regards,

Sindhu

Re: CBO for Hive over hbase

michelsumbul — Tue, 21 Jun 2016 17:05:16 GMT

Thanks for the info. Then what will be your best advice to improve performance of hive over hbase?

Re: CBO for Hive over hbase

tyu — Tue, 21 Jun 2016 20:25:24 GMT

Hive using MR-over-hbase-snapshots would be a viable solution.

Perform a snapshot in hbase, then use hive to directly read from the underlying HFiles.