Support Questions
Find answers, ask questions, and share your expertise

Spark DataFrame to Teradata export - Cannot convert String to varchar

Highlighted

Spark DataFrame to Teradata export - Cannot convert String to varchar

Hi,

I am reading a table in Teradata into Spark and encoding to a custom POJO. There are some fields declared as String. When writing back to Teradata I get the following error:

Exception in thread "main" java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.26] [Error 3706] [SQLState 42000] Syntax error: Data Type "TEXT" does not match a Defined Type name.

String is getting mapped to TEXT instead of varchar. I have tried changing the Teradata JDBC versions yet get the same error.

Anyone knows how to force it to save as varchar instead of TEXT?

Thanks.

3 REPLIES 3
Highlighted

Re: Spark DataFrame to Teradata export - Cannot convert String to varchar

New Contributor

Hi,

Got the same issue, have you been able to resolve?

com.teradata.jdbc.jdbc_4.util.JDBCException: [Teradata Database] [TeraJDBC 14.00.00.21] [Error 3706] [SQLState 42000] Syntax error: Data Type "TEXT" does not match a Defined Type name.

Thanks

Highlighted

Re: Spark DataFrame to Teradata export - Cannot convert String to varchar

@Dima Avramov Yes, referred this pull request https://github.com/apache/spark/pull/13359

We need to write a custom Teradata dialect and force String type to varchar.

The same was done when using Oracle till few months back http://stackoverflow.com/questions/31287182/writing-to-oracle-database-using-apache-spark-1-4-0

Let me know if this works for you.

Highlighted

Re: Spark DataFrame to Teradata export - Cannot convert String to varchar

New Contributor

@Saisubramaniam Gopalakrishnan

Thanks, I think i am going to use sqoop for now, seems to work fine.

Do you know if they resolved the issue in spark 2.0 by chance?