We had to add this to custom hive-site (custom hive-env does not work). Once added to custom hive-site, this worked fine. James ... View more

@Benakaraj KS Go to 2.6.3 and set spark.sql.hive.convertMetastoreOrc=true and spark.sql.orc.enabled=true. You'll get a 3x 🙂 ... View more

@Constantin Stanca ... View more

i forgot to update this. Actually, the LLAP and the cache was setup correctly. In the queries taken from the Teradata, each of the 200k queries performs several joins and one of the joins was on a column that was a null. So, result set was null. Side benefit though was a nice gain in knowledge on all the nobs/levers of this product. And it is a very nice product. My experience is in Teradata/Netezza/Hawq and I've found LLAP to be a clear winner in replacing those. Very fast, very robust. Couple of Notes on the things that mattered: -Use ORC (duh) -Get Partitioned by / Clustered by right (use hive --orcfiledump as shown in Sergey's slides to make sure you get 100k records per orc file) -Get number of nodes / appmasters right -Use local Metastore -Get heap's right -Get yarn capacity scheduler settings right -Increase executors -(probably not supported) clone IS2 jvm's for more throughput if needed. Pull-up hive ui or grafana and sit back, smile and enjoy watching the transactions fly. ... View more

I agree. HDP 2.6 is very solid. We have thoroughly tested against 200k queries/hr (ootb is about 40k/hr, need to go to local metastore, increase appmaster's, etc. to increase). HDP 2.5.3 was solid too, though technically LLAP was tech preview in that release. ... View more

@Benjamin Leonhardi , on slide 24 you notate that a small stripe size indicates a memory problem during load. Do you know what memory problem that would be? I have ~ 3500 records on the stripe and was just wondering where I should look. Thanks! ... View more

@Constantin Stanca yes, in-memory cache per daemon = 12288 (Mb) ... View more

yes - it is in advanced hive-interactive-site. Also notice, there is no set-recommended button (blue 3/4 circle arrow). Thanks again Constantin. ... View more

I should also add everything looks correct, starts correctly and runs well. It just isn't using the io cache. I do have millions of these messages in yarn timeline server log, but from looking at the orc writer code it doesn't look like orc writers uses timeline server: 2017-02-21 00:20:36,581 WARN timeline.EntityGroupFSTimelineStore (LogInfo.java:doParse(207)) - Error putting entity: dag_1487651061826_0016_921 (TEZ_DAG_ID): 6 Other than that, everything is clean. Thanks again, I appreciate it. ... View more

hivellapiomemorymode.png Yeah, I thought so too. It was literally blank (see screenshot of first ambari version post enabling interactive query). Also odd was the hive.llap.daemon.vcpus.per.instance, it was a variable rather than the number. This generated a WARN at start ("WARN conf.HiveConf: HiveConf hive.llap.daemon.vcpus.per.instance expects INT type value"). So I put 32 in for the vcpu's and I put cache for hive.llap.io.memory.mode. They saved correctly to hive-site.xml. The third anamoly is I get is a message that says "WARN conf.HiveConf: HiveConf of name hive.llap.daemon.allow.permanent.fns does not exist". This comes from ambari and I can remove from hive-site.xml, but ambari will put it back. Everything else was as I would expect. hive.llap.io.enabled is true. According to Grafana, I have a 12Gb cache, but there isn't anything in it. Thanks for your help, I appreciate it.. ... View more

yes, typo in post. it is hive.llap.io.memory.mode=cache Thanks! ... View more

@Prasanth Jayachandran @Serge Dyshlevyi ... View more

Update for those out there thinking of answer, (and a cheat sheet for those browsing of good parameters to set - I have tested each individually and have response time down to 630ms and throughput to 40 queries/sec for a 1Tb dataset, but need to get the iocache going to get to the next level and improve throughput (q/sec)): The io cache is created (12Gb), but is never used. I am using OOTB LLAP settings except for below changes, all of which were positive in driving down response time and increasing throughput, except a few were neutral: -increased mem for yarn, llap heap, llap concurrency -disabled CBO (it's adding time to query during creating and submitting plan portion of query) -set hive.llap.iomemory.mode=cache (OOTB this setting was null) -increased /proc/sys/net/core/somaxconn to 4096 -increase metastore heap -increased yarn threads (yarn.resourcemanager.amlauncher.thread-count) -increased Yarn Resource Manager heartbeat interval (tez.am-rm.heartbeat.interval-ms.max) -set hive.llap.daemon.vcpus.per.instance to 32, previously it is unrecognized variable (ambari bug) and gave message in ambari startup of "WARN conf.HiveConf: HiveConf hive.llap.daemon.vcpus.per.instance expects INT type value" -added additional HS2 instance. -added the 4 hive-conf options (hive.execution.mode, hive.llap.execution.mode, hive.lllap.io.enabled, hive.llap.daemon.service.host.) as additional properties in HS2 custom site. -increased HDFS heap Version of HDP is 2.5.3.0-37 Version of Hive is 2.1.0.2.5.3.0-37 ... View more

ah, thanks Ted ... View more