Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Sqoop insert not dynamic partition

Sqoop insert not dynamic partition

Contributor

Hi All

   I tried run below statement but not able to do dynamic partition into Hive. Sqoop only support static partition?

sudo -u cloudera-scm sqoop import --connect jdbc:mysql://host/xxran --fields-terminated-by , --escaped-by \\ --enclosed-by '\"' --username apps --password-file /password.file \
--query "select empno ,ename ,mgr ,hiredate ,sal ,comm from xxran.emp WHERE "'$CONDITIONS'" " \
--split-by empno \
--hive-import \
--create-hive-table \
--hive-table xxran.emp_job \
--hive-partition-key job \
--target-dir '/xxjob' 

Below statement work but it is static partition 

sudo -u cloudera-scm sqoop import --connect jdbc:mysql://host/xxran --fields-terminated-by , --escaped-by \\ --enclosed-by '\"' --username apps --password-file /password.file \

--query "select  empno ,ename ,mgr ,hiredate ,sal ,comm from xxran.emp WHERE "'$CONDITIONS'" " \

--split-by empno \

--hive-import \

--create-hive-table \

--hive-table xxran.emp_job_dynamic \

--hive-partition-key job \

--hive-partition-value 'CLERK' \

--target-dir '/XXJOB'

1 REPLY 1

Re: Sqoop insert not dynamic partition

Champion

HCatalog integration has been added as part of 1.4.4 sqoop release. 

 

You can use dynamic partition , where the partitioning keys are specified in Hive and the corresponding values are coming from Database columns instead of hard coding technique that we do in  static partitioning.

 

Would you consider using Hive Bucketing for decomposing data sets in to more manageable parts. 

 

http://sqoop.apache.org/docs/1.4.4/SqoopUserGuide.html#_hcatalog_background

Don't have an account?
Coming from Hortonworks? Activate your account here