Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Sqoop import data to Hdfs from database with column mapping as parquet

Highlighted

Sqoop import data to Hdfs from database with column mapping as parquet

New Contributor

We want to load data to hdfs from database with sqoop. In hdfs , Data must be parquet. When we executed command1 like  under below  without column mapping load column as String. But there is Decimal(20,2) data   in column  in database , so we want to column mapping like as source. We tried some scenario  like under below.

 

 

In database (

columnA   NUMBER            10      0             

columnB   NUMBER            20                    2             

)

 

Command1;

sqoop import

--connect jdbc:oracle:thin:@host:port/dbname 

--username usname --password-file passfile 

--query "select columnA,columnB from dbschema.dbname where \$CONDITIONS" 

--target-dir 'a/b/c'

 --num-mappers 1

--compress --compression-codec org.apache.hadoop.io.compress.SnappyCodec 

--as-parquetfile 

 

 

Scenario1

sqoop import

--connect jdbc:oracle:thin:@host:port/dbname 

--username usname --password-file passfile 

--query "select columnA,columnB from dbschema.dbname where \$CONDITIONS" 

--target-dir 'a/b/c'

 --num-mappers 1

--compress --compression-codec org.apache.hadoop.io.compress.SnappyCodec 

--map-column-hive 'columnA=BIGINT,columnB=DECIMAL(20%2C2)'

--hive-database x --hive-table y  --as-parquetfile  --hive-import

 

In this scenario , Command was executed but result was not parquet file result was text file and result file path not a/b/c, path was /user/hive/warehouse/x.db/y.

 

 

 

Scenario 2

sqoop import

--connect jdbc:oracle:thin:@host:port/dbname 

--username usname --password-file passfile 

--query "select columnA,columnB from dbschema.dbname where \$CONDITIONS" 

--target-dir 'a/b/c'

 --num-mappers 1

--compress --compression-codec org.apache.hadoop.io.compress.SnappyCodec 

--map-column-java 'columnA=Long,columnB=java.math.BigDecimal (20%2C2)'

--as-parquetfile 

 

 

We got result like “ERROR orm.ClassWriter: No ResultSet method for Java type java.math.BigDecimal(20,2)”

 

 

-Also same scenario 2 with  --map-column-java 'columnA=Long,columnB=java.math.BigDecimal’

 

We got result “ERROR tool.ImportTool: Import failed: Cannot convert to AVRO type java.math.BigDecimal”

 

- Also same scenario 2 with  --map-column-java 'columnA=Long,columnB= BigDecimal’

We got result like “ERROR orm.ClassWriter: No ResultSet method for Java type BigDecimal”

 

- Also same scenario 2 with  --map-column-java 'columnA=Long,columnB= Decimal’

We got result like “ERROR orm.ClassWriter: No ResultSet method for Java type Decimal”

 

 

We want to transfer number 20,2 as Decimal (20,2) to HDFS and also file format have to be parquet .

 

So is there any way to load data to hdfs with map-column as parquet file?