- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Insert records into SQL Server using Spark SQL (version 2) with JDBC
- Labels:
-
Apache Spark
Created 10-12-2017 07:54 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am trying to load records into MS SQL SERVER through Spark 2 using Spark SQL and JDBC connectivity. The program compiled successfully. Also ran without any errors. But records are not inserted into SQL Server. Below I mentioned the code
Spark Code Block:
LeadsDF.write .format("jdbc") .option("url", "jdbc:sqlserver://localhost:1433;databaseName=LeadTester") .option("dbtable", TestTable) .option("user", "XXX") .option("password", "XXXXX") .mode(SaveMode.Append)
SBT Dependencies:
name := "EDWETL" version := "1.0" scalaVersion := "2.11.8" libraryDependencies += "com.microsoft.sqlserver" % "mssql-jdbc" % "6.1.0.jre8" %"provided" libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.0.2" libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.0.0" libraryDependencies += "com.thetransactioncompany" % "java-property-utils" % "1.8" mainClass in (Compile,run) := Some("ssss.HBaseToSQLLoader.Main")
Spark-Submit:
spark-submit \
--class ssss.HBaseToSQLLoader \
--verbose \
--master local \
--driver-class-path mssql-jdbc-6.2.1.jre8.jar \
--num-executors 2 \
--driver-memory 512m \
--executor-memory 512m \
--executor-cores 1 \
/home/maria_dev/Mahe/edwetl_Main.jar
Created 10-30-2017 02:07 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This problem got solved after adding "Auto commit" to true. Below is the updated code.
val driverClass = "com.microsoft.sqlserver.jdbc.SQLServerDriver" val connectionProperties = new java.util.Properties() connectionProperties.setProperty("Driver", driverClass) connectionProperties.setProperty("AutoCommit", "true")