Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive alter table concatenate behaves oddly

Hive alter table concatenate behaves oddly

I have dozens of tables with daily partitions, some of which require concatenation after creation, some of which don't. I'm not sure what to expect when I call concatenate on these partitions. Should it produce (bytecount/blocksize) files of just under the blocksize? Should it produce (square root of line count) files of indeterminate size? Is there a way to tune it?

 

Specifically, I'm trying to reduce my small file problem, but I don't want to call concatenate on partitions if it won't actually do anything.

 

Thanks in advance.