Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive partitioning for 1lac partition


Hive partitioning for 1lac partition

New Contributor

We are having large amount of data in AWS redshift. In all tables there is one column (let's say columnX). We want to store this data in S3 in such a way that for each value of columnX there will be separate file in S3. I tried this with hive partitioning (Unloaded data into redshift, using hive dynamic partitioning to split data by using columnX as partition), but it is taking too long to split data. There will be about 1 lac different values possible for columnX.

Can some one please suggest solution here ?

Don't have an account?
Coming from Hortonworks? Activate your account here