Reply
Highlighted
Explorer
Posts: 6
Registered: ‎02-03-2019

Sqoop directly to Hive advantages?

Hi,

 

Is there is any advantage in importing data from DB2 to hive directly using Sqoop compared to importing to HDFS first and then loading to Hive table? Which is recommeded or preferred approach?

 

Thanks

Posts: 1,903
Kudos: 435
Solutions: 305
Registered: ‎07-31-2013

Re: Sqoop directly to Hive advantages?

Sqoop's import to Hive is an extension of its import to HDFS (i.e. the Hive part is done after its regular HDFS import work), so if your formats are already acceptable and do not need further transformation you can do it as part of the Sqoop step directly.

Sqoop also supports (via its HCatalog options) inserting into partitioned tables via dynamic partitioning if you'll require that.
Explorer
Posts: 6
Registered: ‎02-03-2019

Re: Sqoop directly to Hive advantages?

[ Edited ]

Thanks for the response.

I want to store them as parquet format. Sqoop supports parquet but hive does not recognize those parquet files. So in both options, I have to sqoop them in text format and then move to another hive table in parquet format.

 

option 1 - sqoop to hdfs in text format (hdfs location is hive external table). Then insert into hive table with parition and paquet format

 

option 2 - sqoop to hive internal table in text format, iinsert into hive table with parition and paquet format

 

Based on your response, hive part is done after import to hdfs, I think I can avoid that extra work (time) with option 1. Agree?

 

Posts: 1,903
Kudos: 435
Solutions: 305
Registered: ‎07-31-2013

Re: Sqoop directly to Hive advantages?

Could you elaborate on 'does not recognize it'? If its just an options
problem but the files appear OK if you try loading them manually into Hive,
then you can import to HDFS as Parquet and run a simple LOAD DATA INPATH
statement to move them to your Hive table as-is.

If there's a deeper inconsistency that requires transformation from text to
get it right, then both options seem fine.