Write to Hive from Executors Fails

[#1]

Hi,

Can I write to hive table from executors?

Pseudo Code Scenario

---------------------------------------

I have created a dataframe from sql:

val sqlStr = "Select * from TableA where ... "

val myDF = sparkSession.sql(sqlStr)

this following part fails:

myDF.foerachpartition{

    rows => rows.foreach{

         row => {

                      val results = MyFunc(row.getAs[String]("Col_a"), ....)

                      val insSql = "Insert into myTbale_b values (results.A, results.B...)"

                      sparkSession.sql(sqlStr) ... this call fails

           }

    }

}

I understand the foreach spawns the task across executors...

the executors are not able to write to hive.... how can I get around this

Env:

Spark 2.4

CDH 6.2

Scala 2.11

[#2]

Does it mean we can access sparksession within a foreachpartition construct.

How can one get around this