Reply
Highlighted
New Contributor
Posts: 2
Registered: ‎12-14-2017

Sqoop import data to Hdfs from database with column mapping as parquet

We want to load data to hdfs from database with sqoop. In hdfs , Data must be parquet. When we executed command1 like  under below  without column mapping load column as String. But there is Decimal(20,2) data   in column  in database , so we want to column mapping like as source. We tried some scenario  like under below.

 

 

In database (

columnA   NUMBER            10      0             

columnB   NUMBER            20                    2             

)

 

Command1;

sqoop import

--connect jdbc:oracle:thin:@host:port/dbname 

--username usname --password-file passfile 

--query "select columnA,columnB from dbschema.dbname where \$CONDITIONS" 

--target-dir 'a/b/c'

 --num-mappers 1

--compress --compression-codec org.apache.hadoop.io.compress.SnappyCodec 

--as-parquetfile 

 

 

Scenario1

sqoop import

--connect jdbc:oracle:thin:@host:port/dbname 

--username usname --password-file passfile 

--query "select columnA,columnB from dbschema.dbname where \$CONDITIONS" 

--target-dir 'a/b/c'

 --num-mappers 1

--compress --compression-codec org.apache.hadoop.io.compress.SnappyCodec 

--map-column-hive 'columnA=BIGINT,columnB=DECIMAL(20%2C2)'

--hive-database x --hive-table y  --as-parquetfile  --hive-import

 

In this scenario , Command was executed but result was not parquet file result was text file and result file path not a/b/c, path was /user/hive/warehouse/x.db/y.

 

 

 

Scenario 2

sqoop import

--connect jdbc:oracle:thin:@host:port/dbname 

--username usname --password-file passfile 

--query "select columnA,columnB from dbschema.dbname where \$CONDITIONS" 

--target-dir 'a/b/c'

 --num-mappers 1

--compress --compression-codec org.apache.hadoop.io.compress.SnappyCodec 

--map-column-java 'columnA=Long,columnB=java.math.BigDecimal (20%2C2)'

--as-parquetfile 

 

 

We got result like “ERROR orm.ClassWriter: No ResultSet method for Java type java.math.BigDecimal(20,2)”

 

 

-Also same scenario 2 with  --map-column-java 'columnA=Long,columnB=java.math.BigDecimal’

 

We got result “ERROR tool.ImportTool: Import failed: Cannot convert to AVRO type java.math.BigDecimal”

 

- Also same scenario 2 with  --map-column-java 'columnA=Long,columnB= BigDecimal’

We got result like “ERROR orm.ClassWriter: No ResultSet method for Java type BigDecimal”

 

- Also same scenario 2 with  --map-column-java 'columnA=Long,columnB= Decimal’

We got result like “ERROR orm.ClassWriter: No ResultSet method for Java type Decimal”

 

 

We want to transfer number 20,2 as Decimal (20,2) to HDFS and also file format have to be parquet .

 

So is there any way to load data to hdfs with map-column as parquet file?

Announcements