Add custom UDF on Hive LLAP

[#1]

Hi,

I'm stuck for a while trying to add a custom UDF on Hive LLAP. I followed this link : https://community.cloudera.com/t5/Community-Articles/Creating-custom-udf-and-adding-udf-jar-to-Hive...

But for now without any success...

To resume what I did :

1- uploaded my jar file to the server where HSI is running.

2- modified my conf in Ambari (hive-interactive-env template attribute) and added this line :

export HIVE_AUX_JARS_PATH=$HIVE_AUX_JARS_PATH:<path_to_jar_in_hsi_server>.jar

3-modified my conf in Ambari (auxillary JAR list attribute) and added this line:

<path_to_jar_in_hsi_server>.jar

 4- Restarted LLAP

5- Connected to hsi via beeline and type:

CREATE FUNCTION my_function_name as 'blabla.MyFunction';

Below the results/tests that I did:

In beeline:

show functions ;

==> I can see my function "my_function_name". So for now everything is OK.

Now when I try to use my function in a query like that:

select my_function_name(my_field) from my_table ;

(For information, my function encodes a string. So, in input it takes a string and returns a string encoded in output).

 I have the result:

Error: Error while compiling statement: FAILED: RuntimeException Cannot run all parts of query in llap. Failing since hive.llap.execution.mode is set to only (state=42000,code=40000)

When I just try my function like that:

select my_function("blablabla") ;

It works, I have the result wanted.

What I think is my permanent UDF is only deployed on my HSI server but not on all the LLAP nodes.

I also try to activate this parameters:

set hive.llap.allow.permanent.fns=true ;

Then, I will some security errors:

Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.exec.UDFArgumentException: Unable to instantiate UDF implementation class <my_function>: java.lang.SecurityException

I hope you will have some ideas or solutions.

Thank you in advance for helping me.

I use HDP 2.6

Simon

[#2]

Hi @Simon230 

There are two points which i have to mention here

(a) Temporary functions are not allowed in LLAP

(b)  hive.llap.execution.mode=only will not work with LLAP and custom UDF.  You can use hive.llap.execution.mode=auto or all to run those queries when using a custom UDF. I would suggest you to try with execution mode as auto or all but again it is not the best way to use LLAP.

You can refer the below community article for reference:

https://community.cloudera.com/t5/Support-Questions/How-are-UDF-s-treated-with-Hive-LLAP/td-p/196893... 

Please accept the solution if is is helpful to you.

Thanks!

[#3]

Hi,

Thank you for your response @shobikas 

As explained in my previous post, I don't use temporary functions in LLAP. I try to use permanent function.

I followed this official article https://community.cloudera.com/t5/Community-Articles/Creating-custom-udf-and-adding-udf-jar-to-Hive-... In the example, it seems that it is possible to use a permanent UDF with hive.llap.execution.mode=only.

Thank you,

Simon