Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Query optimization: Where does query optimization happen?

Query optimization: Where does query optimization happen?

New Contributor

Can someone explain if the query optimization happens in the code level or at the database level.

As I understand the code/API accessing the database has no knowledge of the database access patterns and how the data is stored so optimization by database (Hive) makes more sense instead of Spark/Hive(API). But can someone clarify what exactly happens when a query is sent for processing by the Hive driver/SparkSQL to the database. Where does query optimization happen?

Also, I believe DAG creation/resolution is also dependent on the query optimization because based on the optimized query plan the DAG might be shorter or larger? Can someone also add how DAG is handled when query optimization happens?

PS: I am a beginner with Spark/Hadoop/Hive so please correct me if I am understanding it all wrong.

1 REPLY 1

Re: Query optimization: Where does query optimization happen?

Contributor

As far as I'm aware the hive database engine uses the metastore to build the query plan, this would also be where the optimization process happens

I'm curious as to why the specifics of this would impact your use case?

Don't have an account?
Coming from Hortonworks? Activate your account here