Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark DataFrame to Teradata export - Cannot convert String to varchar

Highlighted

Spark DataFrame to Teradata export - Cannot convert String to varchar

New Contributor

Hi,

I am reading a table in Teradata into Spark and encoding to a custom POJO. There are some fields declared as String. When writing back to Teradata I get the following error:

Exception in thread "main" java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.26] [Error 3706] [SQLState 42000] Syntax error: Data Type "TEXT" does not match a Defined Type name.

String is getting mapped to TEXT instead of varchar. I have tried changing the Teradata JDBC versions yet get the same error.

Anyone knows how to force it to save as varchar instead of TEXT?

Thanks.

3 REPLIES 3

Re: Spark DataFrame to Teradata export - Cannot convert String to varchar

New Contributor

Hi,

Got the same issue, have you been able to resolve?

com.teradata.jdbc.jdbc_4.util.JDBCException: [Teradata Database] [TeraJDBC 14.00.00.21] [Error 3706] [SQLState 42000] Syntax error: Data Type "TEXT" does not match a Defined Type name.

Thanks

Re: Spark DataFrame to Teradata export - Cannot convert String to varchar

New Contributor

@Dima Avramov Yes, referred this pull request https://github.com/apache/spark/pull/13359

We need to write a custom Teradata dialect and force String type to varchar.

The same was done when using Oracle till few months back http://stackoverflow.com/questions/31287182/writing-to-oracle-database-using-apache-spark-1-4-0

Let me know if this works for you.

Re: Spark DataFrame to Teradata export - Cannot convert String to varchar

New Contributor

@Saisubramaniam Gopalakrishnan

Thanks, I think i am going to use sqoop for now, seems to work fine.

Do you know if they resolved the issue in spark 2.0 by chance?

Don't have an account?
Coming from Hortonworks? Activate your account here