(SPARK-12823) Cannot create UDF with StructType input -

ram_pratapa — Thu, 14 Jul 2016 06:52:58 GMT

Hi, I am trying create a UDF and use it in dataframe select something like

val selected = jsonRxMap.select(parsePatient($"Patient") ,parseProvider($"Provider"),parsePharmacy($"Pharmacy"))

$"Patient" is StuctureType and I searched google find this SPARK-12823 and I am not sure is there any work around to solve the problem. The goal is that by passing the StructuredType ( struct<FirstName:string,Address1:string,Address2:string,AltID:string,Beeper:string,..) to parsePatient function which returns a unique value to that patient and I can store the patient in a dimention table in hive with ORC format.

Can any one help.

thanks

Ram

Re: (SPARK-12823) Cannot create UDF with StructType input -

ram_pratapa — Mon, 18 Jul 2016 07:09:43 GMT

Hi All, I further researched this issue and found an alternative solution. if you define the function as follows

val parsePatientfun = udf { (thestruct: Row) => thestruct.getAs[String]("City") }

you can get to the fields from StuctureType.

Thanks

Ram

question Re: (SPARK-12823) Cannot create UDF with StructType input - in Archives of Support Questions (Read Only)

(SPARK-12823) Cannot create UDF with StructType input -

Re: (SPARK-12823) Cannot create UDF with StructType input -