Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

(SPARK-12823) Cannot create UDF with StructType input -

avatar
Rising Star

Hi, I am trying create a UDF and use it in dataframe select something like

val selected = jsonRxMap.select(parsePatient($"Patient") ,parseProvider($"Provider"),parsePharmacy($"Pharmacy"))

$"Patient" is StuctureType and I searched google find this SPARK-12823 and I am not sure is there any work around to solve the problem. The goal is that by passing the StructuredType ( struct<FirstName:string,Address1:string,Address2:string,AltID:string,Beeper:string,..) to parsePatient function which returns a unique value to that patient and I can store the patient in a dimention table in hive with ORC format.

Can any one help.

thanks

Ram

1 ACCEPTED SOLUTION

avatar
Rising Star

Hi All, I further researched this issue and found an alternative solution. if you define the function as follows

val parsePatientfun = udf { (thestruct: Row) => thestruct.getAs[String]("City") }

you can get to the fields from StuctureType.

Thanks

Ram

View solution in original post

1 REPLY 1

avatar
Rising Star

Hi All, I further researched this issue and found an alternative solution. if you define the function as follows

val parsePatientfun = udf { (thestruct: Row) => thestruct.getAs[String]("City") }

you can get to the fields from StuctureType.

Thanks

Ram