All,
I am working on writing to RDBMS ( Sql Server ) using Spark from hive, process works with great speed.
But there is big issue, that each tasks until completes does not commits - which utilizes transaction log of the database and can cause impacts to other running jobs.
Need to have some way to commits at regular interval ( 10000 K or so) .
Can someone please suggest how this can be done??
Spark version : 2.2
SQL Server 2016
Thanks
freakabhi