Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

CBO for Hive over hbase

avatar
Rising Star

Hi,

I have Hive running over HBase table. Is it interesting in this case to analyze the table for the CBO? Or is it only interesting whe nyou have hive with orc file?

Thanks in advance,

Michel

1 ACCEPTED SOLUTION

avatar

@Michel Sumbul

CBO is mainly for optimization decisions which reduces the cost of query execution and is independent of storage formats like ORC. Below is some of decisions based on CBO:

  • How to order Join
  • What algorithm to use for a given Join
  • Should the intermediate result be persisted or should it be recomputed on operator failure.
  • The degree of parallelism at any operator (specifically number of reducers to use).
  • Semi Join selection

For details, please refer to below link:

https://cwiki.apache.org/confluence/display/Hive/Cost-based+optimization+in+Hive

Thanks and Regards,

Sindhu

View solution in original post

4 REPLIES 4

avatar

Hello Michel

Right now the Hive plan calculation does not reach out to get Hbases stats so currently no added benefit from the Hbase stats. This being said these are questions that are being worked on in different initiatives, so this will likely change in the future.

avatar

@Michel Sumbul

CBO is mainly for optimization decisions which reduces the cost of query execution and is independent of storage formats like ORC. Below is some of decisions based on CBO:

  • How to order Join
  • What algorithm to use for a given Join
  • Should the intermediate result be persisted or should it be recomputed on operator failure.
  • The degree of parallelism at any operator (specifically number of reducers to use).
  • Semi Join selection

For details, please refer to below link:

https://cwiki.apache.org/confluence/display/Hive/Cost-based+optimization+in+Hive

Thanks and Regards,

Sindhu

avatar
Rising Star

Thanks for the info. Then what will be your best advice to improve performance of hive over hbase?

avatar
Master Collaborator

Hive using MR-over-hbase-snapshots would be a viable solution.

Perform a snapshot in hbase, then use hive to directly read from the underlying HFiles.