question Re: Guidance around setting right number for 'hive.exec.max.created.files' in Support Questions

Guidance around setting right number for 'hive.exec.max.created.files'

schauhan — Fri, 04 Dec 2015 01:10:42 GMT

Hi, I'm looking for some guidance around setting hive.exec.max.created.files. Is there a formula or ratio to follow ahead of time to find a right number for my queries?

Re: Guidance around setting right number for 'hive.exec.max.created.files'

nsabharwal — Fri, 04 Dec 2015 09:23:26 GMT

@schauhan I could not find the answer. Really good question. @gopal @yzhang

Re: Guidance around setting right number for 'hive.exec.max.created.files'

yzhang1 — Fri, 04 Dec 2015 10:05:42 GMT

The main purpose of it is to prevent overloading hdfs? If a query generates more than default #100000, it is better user examine the query and see why so. maybe the query is generating too many too small files.

Re: Guidance around setting right number for 'hive.exec.max.created.files'

sachin_mca25 — Thu, 10 Dec 2015 01:07:28 GMT

are you facing this issue, while trying to load data in a large table.

Re: Guidance around setting right number for 'hive.exec.max.created.files'

gopalv — Wed, 16 Dec 2015 04:55:07 GMT

This is usually a symptom of the problem - general recommendation is to turn on

hive.optimize.sort.dynamic.partition=true;

to prevent partitioning+bucketing from blowing up HDFS file counts.

Re: Guidance around setting right number for 'hive.exec.max.created.files'

aervits — Wed, 03 Feb 2016 23:34:39 GMT

@schauhan are you still having issues with this? Can you accept best answer or provide your own solution?