How are UDF's treated with Hive LLAP?

[#1]

What is the expected/intended behavior with Hive LLAP when using UDF's like ESRI Geo-spatial queries? In the Hive wiki there is mentioned that only 'blessed' UDF's are accepted in LLAP execution.

[#2 · Accepted solution]

LLAP localizes all permanent functions when you restart it - temporary functions aren't allowed into LLAP, since that has potential for conflicting between users.

So expect "add jar" to be a problem, but "create function ... " with an HDFS location to handle something like the ESRI udfs (I recommend creating an esri DB and naming all udfs as esri.ST_Contains etc).

The LLAP decider module should throw an error if you try to use a temporary UDF (when llap.execution.mode is "only", the default).

LLAP can run some of those queries in mixed mode (i.e all mappers run as Tez tasks, all reducers run in LLAP etc), but it's not the best way to use LLAP.

[#3]

Excellent info @gopal, thx!

[#4]

Can someone help me understand what is a "blessed" UDF ?

[#5]

So I was scratching my head a lot too on finding out what "blessed" UDF is supposed to mean. Luckily I was able to make UDFs work on LLAP.

Some findings:

• Even if you're able to create UDFs using Hive shell (non-LLAP), invoking the functions on LLAP mode, will not work. You can invoke functions on non-LLAP sessions only.
• Executing `create function` scripts using LLAP connection(JDBC) and then invoking them immediately will not work. They won't even show when doing a `show functions;` command.

Below are the steps I did to make it work on LLAP:

1. Write drop & create functions using JDBC and execute the same using Hive LLAP jdbc connection

• After executing above java application, you will not see your functions created yet (if you'd check it on Hive shell using `show functions;`)

2. Restart HiveServer2 Interactive and HiveServer2 on Ambari as shown:

3. After successful restart, connect to Hive shell and do a 'show functions;'. Voila! Your UDFs now appear as you wish.

4. After that, you should already be able to invoke your UDFs in LLAP mode

Above steps worked for me using HDP 3

HTH,

Kenneth