Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How can I use UDF at Hive on Spark?

avatar
Contributor

I use hive on spark.

I made UDF.  Jar file name is 'hive-udf-20190701.jar'

 

I set hive configuration(Hive Service Advanced Cofniguration snipped(Safety Value) for hive-site.xml.

hive.reloadable.aux.jars.path
/usr/local/bigdata/hive-udf

 

I upload jar file to HiveServer2 filesystem directory.

/usr/local/bigdata/hive-udf-20190701.jar

I create function on Hue.

reload;
drop temporary function udf_map_tojson;
create temporary function udf_map_tojson as 'bigdata.hive.udf.MapToJsonString';

I test UDF.

select udf_map_tojson(str_to_map("k1:v1,k2:v2"));

 

But, Exception raise.

Error while processing statement: FAILED: Execution Error, 
return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask.
Spark job failed due to: Job aborted due to stage failure:
Aborting TaskSet 3.0 because task 0 (partition 0)
cannot run anywhere due to node and executor blacklist.
Most recent failure: Lost task 0.0 in stage 3.0 (TID 3, worker09.example.com, executor 1):
UnknownReason Blacklisting behavior can be configured via spark.blacklist.*.

What am I wrong?

 

1 ACCEPTED SOLUTION

avatar
Super Guru
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
2 REPLIES 2

avatar
Super Guru
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Contributor

Thanks you.

Yes, It is not problem of UDF.

I have 2 HiveServer2 server host.

I have been register udf only 1 server host.

Maybe. It is reason.