question Re: How can I use UDF at Hive on Spark? in Support Questions

How can I use UDF at Hive on Spark?

avengers — Fri, 16 Sep 2022 14:28:57 GMT

I use hive on spark.

I made UDF. Jar file name is 'hive-udf-20190701.jar'

I set hive configuration(Hive Service Advanced Cofniguration snipped(Safety Value) for hive-site.xml.

hive.reloadable.aux.jars.path
/usr/local/bigdata/hive-udf

I upload jar file to HiveServer2 filesystem directory.

/usr/local/bigdata/hive-udf-20190701.jar

I create function on Hue.

reload;
drop temporary function udf_map_tojson;
create temporary function udf_map_tojson as 'bigdata.hive.udf.MapToJsonString';

I test UDF.

select udf_map_tojson(str_to_map("k1:v1,k2:v2"));

But, Exception raise.

Error while processing statement: FAILED: Execution Error, 
return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. 
Spark job failed due to: Job aborted due to stage failure: 
Aborting TaskSet 3.0 because task 0 (partition 0) 
cannot run anywhere due to node and executor blacklist. 
Most recent failure: Lost task 0.0 in stage 3.0 (TID 3, worker09.example.com, executor 1): 
UnknownReason Blacklisting behavior can be configured via spark.blacklist.*.

What am I wrong?

Re: How can I use UDF at Hive on Spark?

EricL — Fri, 05 Jul 2019 05:01:02 GMT

Hi @avengers,

The error does not seem to be related to the UDF you used, as there is no reference to it from the error message. Have you tried to run other queries to confirm?

Is there other exception error messages in the application log to indicate why this particular executor failed to run the job?

Cheers
Eric

Re: How can I use UDF at Hive on Spark?

avengers — Mon, 05 Aug 2019 03:54:21 GMT

Thanks you.

Yes, It is not problem of UDF.

I have 2 HiveServer2 server host.

I have been register udf only 1 server host.

Maybe. It is reason.