Support Questions

Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

Insert records into SQL Server using Spark SQL (version 2) with JDBC

Contributor

I am trying to load records into MS SQL SERVER through Spark 2 using Spark SQL and JDBC connectivity. The program compiled successfully. Also ran without any errors. But records are not inserted into SQL Server. Below I mentioned the code

Spark Code Block:

LeadsDF.write  
.format("jdbc")  
.option("url", "jdbc:sqlserver://localhost:1433;databaseName=LeadTester")  
.option("dbtable", TestTable)  
.option("user", "XXX")  
.option("password", "XXXXX")  .mode(SaveMode.Append)


SBT Dependencies:

name := "EDWETL"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies += "com.microsoft.sqlserver" % "mssql-jdbc" % "6.1.0.jre8" %"provided"
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.0.2"
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.0.0"
libraryDependencies += "com.thetransactioncompany" % "java-property-utils" % "1.8"
mainClass in (Compile,run) := Some("ssss.HBaseToSQLLoader.Main")

Spark-Submit:

spark-submit \

--class ssss.HBaseToSQLLoader \

--verbose \

--master local \

--driver-class-path mssql-jdbc-6.2.1.jre8.jar \

--num-executors 2 \

--driver-memory 512m \

--executor-memory 512m \

--executor-cores 1 \

/home/maria_dev/Mahe/edwetl_Main.jar

1 REPLY 1

Contributor

This problem got solved after adding "Auto commit" to true. Below is the updated code.

val driverClass = "com.microsoft.sqlserver.jdbc.SQLServerDriver"
val connectionProperties = new java.util.Properties()
connectionProperties.setProperty("Driver", driverClass)
connectionProperties.setProperty("AutoCommit", "true")
Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.