Reply
Highlighted
Posts: 519
Topics: 14
Kudos: 91
Solutions: 45
Registered: ‎09-02-2016

Using Parquet in sqoop import automatically converts the datatypes in Hive table

[ Edited ]

Hi

 

We are planning to implement the Parquet logic in Sqoop Import for the existing process. But it has converted some column datatype in Hive table automatically as follows:

 

Table_Name	Column Name			Data_Type(Before parquet)	Data_Type(After Parquet)		
Table1		actiontime			string						bigint						
Table1		createdate			string						bigint						
Table1		action				string						string						
Table1		studentid			double						string						

Sqoop import is for our staging load. if parquet automatically converts the data type this will impact the sub-sequent codes, so we want to maintain the Source Data_Type in Hive as it is... How to customize the datatype in sqoop import when using 'parquet'?

 

 

Thanks

Kumar

 

Champion
Posts: 761
Registered: ‎05-16-2016

Re: Using Parquet in sqoop import automatically converts the datatypes in Hive table

Would you consider using --map-coloum-hive <mapping>  in your sqoop import exeution. 

example

sqoop import ... --map-column-hive actiontime=String,

 

Announcements
New solutions