orc small files Concatenate in Hive

[#1]

Hi,

I have an ORC table which updates every 5 min in different date partitions, i want to run the CONCATENATE alter table command, but how do i run it on all partitions at once? with the below command i could do only on single partition:

ALTER TABEL tablename PARTITION (dt=20180109) CONCATENATE;

Thanks.

[#2]

I don't think there is a way to do it at once for all partitions, best you could is to specify multiple partitions like ALTER TABLE tableName PARTITION(dt=20180109, dt=20180110..) CONCATENATE. Please note that there are known issues with ALTER TABLE CONCATENATE in versions earlier than HDP 2.6 and it is not recommended to run CONCATENATE.

[#3]

@vgarg

Thanks for the answer.

I use 2.5.3 HDP, when i use concatenate once on the partition with many files, it only concatenates 1 or a few files each time, i have to do it multiple times to concatenate all to one large file.. was this an issue as well? could be please direct me to the issues of the concatenate in earlier versions?

[#4]

@vgarg / @PJ : I am using HDP 2.6 but still I am facing the same issue.

[#5]

See https://community.hortonworks.com/questions/212611/hivepartitionssmall-filesconcatenate.html