Hi,   I'm writing to you because I have some trouble or misunderstanding with Hive LLAP. LLAP is slow when executing queries and I feel like LLAP is not well configured. I checked the jmx metrics of my LLAP daemons and I saw that :   "ExecutorTotalRequestsHandled" : 89065, "ExecutorTotalSuccess" : 22893, "ExecutorTotalFailed" : 58, "ExecutorTotalKilled" : 21724, "ExecutorTotalEvictedFromWaitQueue" : 43678, "ExecutorTotalRejectedRequests" : 406234, "ExecutorTotalPreemptionTimeToKill" : 73450159, "ExecutorTotalPreemptionTimeLost" : 1083694998,   Each deamon has the same behavior.   Most of executed tasks are killed or evicted (only 25% of sucess tasks...) and I think this is one of the reason of the slowness.   Below my configuration : - 75 nodes reserved for LLAP (node labelling). - 280 GB - 40 CPUs per node. - 220 GB - 36 CPU per daemon. - Daemon : 144 GB for the heap, 60 GB for the cache, 16 GB for the headroom. - hive.tez.container.size = 4 G - tez.am.resource.memory.mb = 12 G - hive.llap.io.threadpool.size = 36   Does this configuration look well for you ?   Also, when I check the execution of some queries, I can see that :   Map 1: 855(+40426,-1167)/41281 Reducer 2: 0/1009   It means that "Map 1" has 41281 tasks map, 855 are completed and 40426 are running (and 1167 are pending) but how it that possible because I have 75 daemon and 36 CPUs per deamon (75*36 = 2700), So I should have at most 2700 tasks running (1 CPU per task), no ?   What do you think ? Any ideas, suggestions or questions would be useful.   Thanks a lot for your help.   ... View more

Hi,   I'm stuck for a while trying to add a custom UDF on Hive LLAP. I followed this link : https://community.cloudera.com/t5/Community-Articles/Creating-custom-udf-and-adding-udf-jar-to-Hive-LLAP/ta-p/246598 But for now without any success...   To resume what I did :   1- uploaded my jar file to the server where HSI is running.   2- modified my conf in Ambari (hive-interactive-env template attribute) and added this line : export HIVE_AUX_JARS_PATH=$HIVE_AUX_JARS_PATH:<path_to_jar_in_hsi_server>.jar   3-modified my conf in Ambari (auxillary JAR list attribute) and added this line: <path_to_jar_in_hsi_server>.jar  4- Restarted LLAP   5- Connected to hsi via beeline and type:   CREATE FUNCTION my_function_name as 'blabla.MyFunction';   Below the results/tests that I did: In beeline: show functions ; ==> I can see my function "my_function_name". So for now everything is OK.   Now when I try to use my function in a query like that: select my_function_name(my_field) from my_table ; (For information, my function encodes a string. So, in input it takes a string and returns a string encoded in output).  I have the result: Error: Error while compiling statement: FAILED: RuntimeException Cannot run all parts of query in llap. Failing since hive.llap.execution.mode is set to only (state=42000,code=40000)   When I just try my function like that: select my_function("blablabla") ; It works, I have the result wanted.   What I think is my permanent UDF is only deployed on my HSI server but not on all the LLAP nodes.   I also try to activate this parameters: set hive.llap.allow.permanent.fns=true ; Then, I will some security errors: Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.exec.UDFArgumentException: Unable to instantiate UDF implementation class <my_function>: java.lang.SecurityException   I hope you will have some ideas or solutions.   Thank you in advance for helping me.   I use HDP 2.6   Simon ... View more