Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Write to Hive from Executors Fails

Write to Hive from Executors Fails

Explorer

Hi,

Can I write to hive table from executors?

 

Pseudo Code Scenario

---------------------------------------

I have created a dataframe from sql:

val sqlStr = "Select * from TableA where ... "

val myDF = sparkSession.sql(sqlStr)

 

this following part fails:

myDF.foerachpartition{

    rows => rows.foreach{

         row => {

                      val results = MyFunc(row.getAs[String]("Col_a"), ....)

                      val insSql = "Insert into myTbale_b values (results.A, results.B...)"

                      sparkSession.sql(sqlStr) ... this call fails

           }

    }

}

 

I understand the foreach spawns the task across executors...

the executors are not able to write to hive.... how can I get around this

 

Env:

Spark 2.4

CDH 6.2

Scala 2.11

 

1 REPLY 1
Highlighted

Re: Write to Hive from Executors Fails

Explorer

Does it mean we can access sparksession within a foreachpartition construct.

How can one get around this

Don't have an account?
Coming from Hortonworks? Activate your account here