Reply
New Contributor
Posts: 1
Registered: ‎08-06-2013

Dynamic partition

Hi

 

We are using dynamic partition on 250GB table. The issue is query is working on small table but is not working on huge data set like more then 100 GB . We are following the Apache wiki and rewrite the query using " distributed by " but still didnt help . The query is failing in the reducer phase . Any Guidence or idea guys .....

 

 

Regards

Rohit

Cloudera Employee
Posts: 62
Registered: ‎07-29-2013

Re: Dynamic partition

Hi Rohit,

Thanks for posting. I can't say much without knowing the actual errors but it seems like you may need to adjust some properties related to dynamic partitioning.

 

I would strongly suggest reading https://cwiki.apache.org/confluence/display/Hive/Tutorial#Tutorial-DynamicpartitionInsert

 

The three properties you probably want to tune are:

hive.exec.max.dynamic.partitions.pernode

hive.exec.max.dynamic.partitions

hive.exec.max.created.files

 

They defaults are kept intentionally low to conservatively present JVM heap size errors.

 

Also, I am assuing you are already setting the mode correctly by doing something like:

set hive.exec.dynamic.partition.mode=nonstrict;

 

Expert Contributor
Posts: 63
Registered: ‎08-06-2013

Re: Dynamic partition

The query is failing in the reducer phase.

 

What is the error message in the reducer phase?