Support Questions
Find answers, ask questions, and share your expertise

Dynamic partition with sqoop

Highlighted

Dynamic partition with sqoop

New Contributor

I read from sqoop documentation

http://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html#_hcatalog_background

23.6. Support for Partitioning

The Sqoop HCatalog feature supports the following table types:

  • Unpartitioned tables
  • Partitioned tables with a static partitioning key specified
  • Partitioned tables with dynamic partition keys from the database result set
  • Partitioned tables with a combination of a static key and additional dynamic partitioning keys

But I am not getting how to use dynamic partition using sqoop.

sqoop import --connect jdbc:mysql://localhost:3306/test
--username root
--password-file file:///home/hdfs/.password
--split-by id
--table test 
--delete-target-dir
--fields-terminated-by ","
--hcatalog-table test
--hcatalog-storage-stanza "stored as orcfile"
--create-hcatalog-table
--hcatalog-partition-keys created_at
--map-column-hive created_at=date,updated_at=date

Getting This error


Either both --hcatalog-partition-keys and --hcatalog-partition-values should be provided or both of these options should be omitted.
and if I use the partition value then it assigns it as static partition

sqoop import --connect jdbc:mysql://localhost:3306/test
--username root
--password-file file:///home/hdfs/.password
--split-by id
--table test 
--delete-target-dir
--fields-terminated-by ","
--hcatalog-table test
--hcatalog-storage-stanza "stored as orcfile"
--create-hcatalog-table
--hcatalog-partition-keys created_at
--hcatalog-partition-value created_at
--map-column-hive created_at=date,updated_at=date

after giving created_at as hcatalog-partition-value it created the directory like apps/hive/warehouse/test/created_at=created_at

it should have used created_at column to create dynamic partition based on data.

Please help me fix this issue

Don't have an account?