Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Control storage files naming for external hive table after merging files using insert overwrite

Control storage files naming for external hive table after merging files using insert overwrite

New Contributor

Hello everyone,

Is it possible to have control over the name convention for files after applying insert overwrite to merge files?

For example; assume external table t with 2 partitions (x and y) that have the following files under these partitions that were generated:

Note here; i am using a file pattern which is x1_y2_[seq] where first two values are partition values.

 

-rw-r--r--   3 hive        hdfs     112511 2019-10-13 19:12 hdfs://HADOOP01/user/datastore/t/x=x1/y=y1/file_x1_y1_1
-rw-r--r--   3 hive        hdfs     112511 2019-10-13 19:12 hdfs://HADOOP01/user/datastore/t/x=x1/y=y1/file_x1_y1_2
-rw-r--r--   3 hive        hdfs     112511 2019-10-13 19:12 hdfs://HADOOP01/user/datastore/t/x=x1/y=y1/file_x1_y1_3

 

and files being pushed based based on multiple thread using external process out of hadoop that generates these files in the corresponding dirs. 

and so i need to merge those files later using insert overwrite.. and for example this would be the result.

 

-rw-r--r--   3 hive        hdfs     337533 2019-10-13 19:12 hdfs://HADOOP01/user/datastore/t/x=x1/y=y1/000001_0

 

 

This should generate 1 file per partition, but with it's own naming like here 000001_0, so my question can i have control on this naming? for example to maintain my original naming like parition1_parition2, so result is x1_x2.

 

I am asking this, as later on i will insert those values to another table that will be used by customer, so i wanted to have some kind of control over the process overall. by comparing values loaded values between what was generated vs what what was pushed to the end tables.

 

Thanks in advance. 

 

Don't have an account?
Coming from Hortonworks? Activate your account here