Member since
09-28-2015
41
Posts
44
Kudos Received
12
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3164 | 04-12-2017 12:19 PM | |
3301 | 01-26-2017 04:38 AM | |
824 | 01-10-2017 10:39 PM | |
2161 | 08-16-2016 07:12 PM | |
14579 | 07-20-2016 06:14 PM |
02-22-2019
09:58 PM
So I was scratching my head a lot too on finding out what "blessed" UDF is supposed to mean. Luckily I was able to make UDFs work on LLAP. Some findings: Even if you're able to create UDFs using Hive shell (non-LLAP), invoking the functions on LLAP mode, will not work. You can invoke functions on non-LLAP sessions only. Executing `create function` scripts using LLAP connection(JDBC) and then invoking them immediately will not work. They won't even show when doing a `show functions;` command. Below are the steps I did to make it work on LLAP: 1. Write drop & create functions using JDBC and execute the same using Hive LLAP jdbc connection After executing above java application, you will not see your functions created yet (if you'd check it on Hive shell using `show functions;`) 2. Restart HiveServer2 Interactive and HiveServer2 on Ambari as shown: 3. After successful restart, connect to Hive shell and do a 'show functions;'. Voila! Your UDFs now appear as you wish. 4. After that, you should already be able to invoke your UDFs in LLAP mode Above steps worked for me using HDP 3 HTH, Kenneth
... View more
07-20-2016
06:14 PM
Looks like your datanodes are dying from too many open files - check the nofiles setting for the "hdfs" user in /etc/security/limits.d/
If you want to bypass that particular problem by changing the query plan, try with
set hive.optimize.sort.dynamic.partition=true;
... View more
07-27-2016
11:16 AM
The cluster is fairly small as its mostly experimental but I have 3 out of the 4 nodes in the cluster that each have 4 vCores and 1GB of memory, with a global YARN minimum memory container size of 256MB. So when you say slots I'm assuming that would translate into 12 slots/containers potentially? i.e. a container representing 1vCore + 256MB. I had assumed that for the resource (CPU/RAM) available in my cluster that the query I'm running on the dataset sizes I'm working with i..e 30-40k records would be more than enough?
... View more
06-04-2016
05:34 AM
ORC is considering adding a faster decompression in 2016 - zstd (ZStandard). The enum values for that has already been reserved, but until we work through the trade-offs involved in ZStd - more on that sometime later this year.
https://issues.apache.org/jira/browse/ORC-46 But bigger wins are in motion for ORC with LLAP, the in-memory format for LLAP isn't compressed at all - so it performs like ORC without compression overheads, while letting the cold data on disk sit around in Zlib.
... View more
10-21-2015
09:47 AM
2 Kudos
@cliu@hortonworks.com This is very helpful benchmarks posted by Amplab. Click
... View more