Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark JDBC changes the columns datatype (Oracle)

Spark JDBC changes the columns datatype (Oracle)

Explorer

Hello,

 

I writing a program that copies data from Hive to an Oracle table.

The copy goes well except for one thing : it changes my Oracle columns' datatype.

 

In oracle I have a schema like that one :


COL1   VARCHAR2(10 CHAR) NOT NULL,
COL2   VARCHAR2(50 CHAR) NOT NULL,
COL3   VARCHAR2(15 CHAR) NOT NULL,
COL4   NUMBER(10)        NOT NULL,
COL5   TIMESTAMP         NOT NULL,
COL6   TIMESTAMP,
COL7   NUMBER(10),
COL8   VARCHAR2(1 CHAR),
COL9   DATE,
COL10  DATE,
COL11  DATE

 

But after Spark has finished the copy, the Oracle schema has changed to this :

 

COL1   VARCHAR2(255 CHAR),
COL2   VARCHAR2(255 CHAR),
COL3   VARCHAR2(255 CHAR),
COL4   NUMBER(10,0),      
COL5   TIMESTAMP(6),     
COL6   TIMESTAMP(6),
COL7   NUMBER(10,0),
COL8   NUMBER(10,0),
COL9   TIMESTAMP(6),
COL10  TIMESTAMP(6),
COL11  TIMESTAMP(6)

 

Even if I change the columns' datatype in Hive, the result is the same.

 

Is it possible to write from Hive to Oracle without Spark modifying the Oracle schema ?

 

Thanks !

2 REPLIES 2

Re: Spark JDBC changes the columns datatype (Oracle)

New Contributor

Hello @adrien555,thanks for your post,


I currently have the same problem, I didn't found any solutions to avoid this problem.

 

Any ideas ?

Re: Spark JDBC changes the columns datatype (Oracle)

Explorer
just cast it back to the correct data type using selectExpr val ConvertedDF = myDF.selectExpr("id","name","cast(age as tinyint) age");
Don't have an account?
Coming from Hortonworks? Activate your account here