Support Questions
Find answers, ask questions, and share your expertise

Write to Hive from Executors Fails

Explorer

Hi,

Can I write to hive table from executors?

 

Pseudo Code Scenario

---------------------------------------

I have created a dataframe from sql:

val sqlStr = "Select * from TableA where ... "

val myDF = sparkSession.sql(sqlStr)

 

this following part fails:

myDF.foerachpartition{

    rows => rows.foreach{

         row => {

                      val results = MyFunc(row.getAs[String]("Col_a"), ....)

                      val insSql = "Insert into myTbale_b values (results.A, results.B...)"

                      sparkSession.sql(sqlStr) ... this call fails

           }

    }

}

 

I understand the foreach spawns the task across executors...

the executors are not able to write to hive.... how can I get around this

 

Env:

Spark 2.4

CDH 6.2

Scala 2.11

 

1 REPLY 1

Explorer

Does it mean we can access sparksession within a foreachpartition construct.

How can one get around this

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.