question Re: Hi, Is there any connector for teradata to spark.We have scenarios to get the data from teradata by using SparkSQl. I am using spark 1.6.0.Please let me know if anyone tired connecting teradata $ spark.Thanks! in Archives of Support Questions (Read Only)

Hi, Is there any connector for teradata to spark.We have scenarios to get the data from teradata by using SparkSQl. I am using spark 1.6.0.Please let me know if anyone tired connecting teradata $ spark.Thanks!

senthilkumarP — Fri, 28 Oct 2016 00:28:19 GMT

Re: Hi, Is there any connector for teradata to spark.We have scenarios to get the data from teradata by using SparkSQl. I am using spark 1.6.0.Please let me know if anyone tired connecting teradata $ spark.Thanks!

jwiden — Fri, 28 Oct 2016 01:48:55 GMT

Sounds like a JDBC connection is in order. There is an api for creating a dataframe from jdbc connection.

jdbc(url: String, table: String, predicates: Array[String], connectionProperties:Properties): DataFrame

The issue with JDBC is reading data from teradata will be much slower compared to HDFS. Is it possible to run a sqoop job to move data to hdfs prior to starting your spark application?

Re: Hi, Is there any connector for teradata to spark.We have scenarios to get the data from teradata by using SparkSQl. I am using spark 1.6.0.Please let me know if anyone tired connecting teradata $ spark.Thanks!

senthilkumarP — Fri, 28 Oct 2016 09:10:09 GMT

Hi Joe,

We want to use spark sql instead sqoop.I tried teradata JDBC driver .Unable to download dependencies .

Thanks

<dependency>
    <groupId>com.teradata.jdbc</groupId>
    <artifactId>terajdbc4</artifactId>
    <version>15.10.00.22</version>
  </dependency>
<dependency>
    <groupId>com.teradata.jdbc</groupId>
    <artifactId>tdgssconfig</artifactId>
    <version>15.00.00.22</version>
  </dependency>

Re: Hi, Is there any connector for teradata to spark.We have scenarios to get the data from teradata by using SparkSQl. I am using spark 1.6.0.Please let me know if anyone tired connecting teradata $ spark.Thanks!

jwiden — Mon, 31 Oct 2016 11:14:57 GMT

Make sure you add the jar to your class path and include it when you run the application.

sc.addJar("yourDriver.jar")

val jdbcDF = sqlContext.load("jdbc", Map(
  "url" -> "jdbc:teradata://<server_name>, TMODE=TERA, user=my_user, password=*****",
  "dbtable" -> "schema.table_name",
  "driver" -> "com.teradata.jdbc.TeraDriver"))

Re: Hi, Is there any connector for teradata to spark.We have scenarios to get the data from teradata by using SparkSQl. I am using spark 1.6.0.Please let me know if anyone tired connecting teradata $ spark.Thanks!

senthilkumarP — Mon, 31 Oct 2016 13:26:06 GMT

Yes ..able to get the tables from teradata .its working fine .Thanks 🙂

Re: Hi, Is there any connector for teradata to spark.We have scenarios to get the data from teradata by using SparkSQl. I am using spark 1.6.0.Please let me know if anyone tired connecting teradata $ spark.Thanks!

fjfbupt — Mon, 20 Mar 2017 19:24:17 GMT

Add jars in the spark-defaults.conf:

spark.driver.extraClassPath /opt/spark/jars/terajdbc4.jar:/opt/spark/jars/tdgssconfig.jar

spark.executor.extraClassPath /opt/spark/jars/terajdbc4.jar:/opt/spark/jars/tdgssconfig.jar

But get the invalid IP for the following commands:

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff} span.s1 {font-variant-ligatures: no-common-ligatures}

scala> val jdbcDF = sqlcontext.load("jdbc", Map("url" -> "jdbc:teradata://****my**:1025/john, TMODE=TERA, user=john, password=pass", "dbtable" -> "john.abc", "driver" -> "com.teradata.jdbc.TeraDriver"))

warning: there was one deprecation warning; re-run with -deprecation for details

2017-03-20.08:14:29.170 TERAJDBC4 ERROR [main] com.teradata.jdbc.jdk6.JDK6_SQL_Connection@1504b493 Connection to 9.26.74.151:1025 Mon Mar 20 08:14:29 EDT 2017 invalid IPv6 address at java.net.InetAddress.getAllByName(InetAddress.java:1169) at java.net.InetAddress.getAllByName(InetAddress.java:1126) at com.teradata.jdbc.jdbc_4.io.TDNetworkIOIF$Lookup.doLookup(TDNetworkIOIF.java:222) at com.teradata.jdbc.jdbc_4.io.TDNetworkIOIF$Lookup.isLiteralIpAddress(TDNetworkIOIF.java:248) at com.teradata.jdbc.jdbc_4.io.TDNetworkIOIF.connectToHost(TDNetworkIOIF.java:335) at com.teradata.jdbc.jdbc_4.io.TDNetworkIOIF.createSocketConnection(TDNetworkIOIF.java:155) at com.teradata.jdbc.jdbc_4.io.TDNetworkIOIF.<init>(TDNetworkIOIF.java:141) at com.terada

Re: Hi, Is there any connector for teradata to spark.We have scenarios to get the data from teradata by using SparkSQl. I am using spark 1.6.0.Please let me know if anyone tired connecting teradata $ spark.Thanks!

AKR — Tue, 12 Nov 2019 14:44:36 GMT

Hi,

We wont provide any connectors for Teradata to spark. but if you want to get data from Teradata into Spark, you can probably use any JDBC driver that Teradata provides.

Thanks

AKR