(SPARK-12823) Cannot create UDF with StructType input -

[#1]

Hi, I am trying create a UDF and use it in dataframe select something like

val selected = jsonRxMap.select(parsePatient($"Patient") ,parseProvider($"Provider"),parsePharmacy($"Pharmacy"))

$"Patient" is StuctureType and I searched google find this SPARK-12823 and I am not sure is there any work around to solve the problem. The goal is that by passing the StructuredType ( struct<FirstName:string,Address1:string,Address2:string,AltID:string,Beeper:string,..) to parsePatient function which returns a unique value to that patient and I can store the patient in a dimention table in hive with ORC format.

Can any one help.

thanks

Ram

[#2 · Accepted solution]

Hi All, I further researched this issue and found an alternative solution. if you define the function as follows

val parsePatientfun = udf { (thestruct: Row) => thestruct.getAs[String]("City") }

you can get to the fields from StuctureType.

Thanks

Ram