@Benjamin Leonhardi I went through your slides and got few questions around that.
Since its related to bucketing, partitioning, I think it makes sense to
continue in the same thread itself.
In dynamic partition (DP) loading, you used the
word standard load. Does that mean setting the number of reducers to 0 or you
mean something else?
You mentioned that larger number of writers and
large number of partitions lead to small files. The number of mappers is based
on the number of blocks and each mapper writes separate files to every
partition. So irrespective of the number of partitions, the large number of
mappers itself can lead to small files. Am I right?
What is the default key used for distribution if
you don’t use the distribute by clause? What is the distribution key when there
are more than 1 partition?
Slide 13 – to enable this kind of load, do you recommend
to just set the number of reducers to be same as the number of partitions? And
any reducer can get data for any partition, which may lead to small files, is
this what you mean(hash conflict)?
Slide 14 – One reducer for each bucket across
all partitions lead to ORC writer memory issues. Why is this the case?
Optimized Dynamic sorted partitioning – one reducer
for each partition and bucket. From above point, there are 5 partitions and 4
buckets then 4 reducers only, but in the optimized case there are 20 reducers.
More the number of reducers, smaller the files are going to be? How can this
solve the small file problem?
Sort by for PPD – ORC index would anyways help
to ignore reading some blocks. But when it comes to reading the block which has
the predicate value, sorting helps performance only when the predicate value is
reached quickly when reading the file. If the value happens to be at the end of
the file then you still end up reading the whole file. So performance
improvement in PPD with Sort by really depends, am I right?
... View more
