Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

(SPARK-12823) Cannot create UDF with StructType input -

Solved Go to solution

(SPARK-12823) Cannot create UDF with StructType input -

Contributor

Hi, I am trying create a UDF and use it in dataframe select something like

val selected = jsonRxMap.select(parsePatient($"Patient") ,parseProvider($"Provider"),parsePharmacy($"Pharmacy"))

$"Patient" is StuctureType and I searched google find this SPARK-12823 and I am not sure is there any work around to solve the problem. The goal is that by passing the StructuredType ( struct<FirstName:string,Address1:string,Address2:string,AltID:string,Beeper:string,..) to parsePatient function which returns a unique value to that patient and I can store the patient in a dimention table in hive with ORC format.

Can any one help.

thanks

Ram

1 ACCEPTED SOLUTION

Accepted Solutions

Re: (SPARK-12823) Cannot create UDF with StructType input -

Contributor

Hi All, I further researched this issue and found an alternative solution. if you define the function as follows

val parsePatientfun = udf { (thestruct: Row) => thestruct.getAs[String]("City") }

you can get to the fields from StuctureType.

Thanks

Ram

1 REPLY 1

Re: (SPARK-12823) Cannot create UDF with StructType input -

Contributor

Hi All, I further researched this issue and found an alternative solution. if you define the function as follows

val parsePatientfun = udf { (thestruct: Row) => thestruct.getAs[String]("City") }

you can get to the fields from StuctureType.

Thanks

Ram