Finding out the hit rate It is possible to determine cache hit rate per node or per query. Per node, you can see hit rate by looking at LLAP metrics view (<llap node>:15002, see General debugging): Per query, it is possible to see LLAP IO counters (including hit rate) upon running the query by setting hive.tez.exec.print.summary=true, which should produce the counters output at the end of the query, for example - empty cache: Some data in cache: Why is the cache hit rate low Consider the data size and the cache size across all nodes. E.g. with a 10 Gb cache on each of the 4 LLAP nodes, reading 1 Tb of data cannot achieve more than ~4% cache hit rate even in the perfect case. In practice the rate will be lower due to effects of compression (cache is not snappy/gzip compressed, only encoded), interference from other queries, locality misses, etc. In HDP 2.X/Hive 2.X, cache has coarser granularity to avoid some fragmentation issues that are resolved in HDP 3.0/Hive 3.0. This can cause considerable wasted memory in cache on some workload, esp. if the table has a lot of strings with a small range of values, and/or is written with smaller compression buffer sizes than 256Kb. When writing data, you might consider ensuring that ORC compression buffer size is set to 256Kb, and set hive.exec.orc.buffer.size.enforce=true (on HDP 2.6, it requires a backport) to disable writing smaller CBs. This issue doesn't result in errors but can make cache less efficient. If the cache size seems sufficient, check relative data and metadata hit rates (see above screenshots). If there are both data and metadata misses, it can be due to other queries caching different data in place of this queries data, or it could be a locality issue. Check hive.llap.task.scheduler.locality.delay; it can be increased (or set to -1 for infinite delay) to get better locality at the cost of waiting longer to launch tasks, if IO is a bottleneck. If metadata hit rate is very high but data is lower, it is likely that cache doesn't fit all the data; so, some data gets evicted, but metadata that is cached with high priority stays in cache. ... View more

This article is a short summary of LLAP-specific causes of query slowness. Given that a lot of the Hive query execution (compilation, almost all operator logic) stay the same when LLAP is used, queries can be slow for non-LLAP-related reasons; general Hive query performance issues (e.g. bad plan, skew, poor partitioning, slow fs, ...) should also be considered when investigating. These issues are outside of the scope of this article. Queries are slow On HDP 2.5.3/Hive 2.2 and below, there’s a known issue where queries on LLAP can get slower over time on some workloads. Upgrade to HDP 2.6.X/Hive 2.3 or higher. If you are comparing LLAP against containers on the same cluster, check LLAP cluster size compared to the capacity used for containers. If there are 27 nodes for containers, and a 3-node LLAP cluster, it’s possible containers will be faster because they can use many more CPUs/memory. Generally, queries on LLAP run just like in Hive (see Architecture). So, the standard Hive performance debugging should be performed, starting with explain, looking at Tez view DAG timeline, etc. to narrow it down. Query is stuck (not just slow) Make sure hive.llap.daemon.task.scheduler.enable.preemption is set to true. Look at Tez view to see which tasks are running. Choose one running task (esp. if there’s only one), go to <its llap node>:15002/jmx view, and search for ExecutorsStatus for this task. If the task is missing or says “queued”, it may be a priority inversion bug. A number were fixed over time in HDP 2.6.X. If the task is running, it is possible that this is a general query performance issue (e.g. skew, bad plan, etc.). You can double check <llap node>:15002/stacks to see if it’s running code. ... View more

This article will (hopefully) be helpful if you are trying to use LLAP/Hive Interactive, but it fails or gets stuck during startup. This is often caused by misconfiguration, esp. w.r.t. sizing, but could also be a bug. Ambari performs 2 major activities when (re)starting Hive Interactive - first it starts LLAP itself on the YARN cluster, then it starts HiveServer2 for Hive Interactive. Usually the problems happen during LLAP startup. This article will help you to pinpoint the problem if there's some bug or non-sizing-related misconfiguration. It will also point at the sizing document where appropriate; the sizing and setup document can be found at https://community.hortonworks.com/articles/149486/llap-sizing-and-setup.html. For general information (e.g. where the logs are) see the parent article. Note: for all the logs, it is sometimes the case that the last error in the log is InterruptedException or some other similar error during component shutdown after something else has already failed. It might help to look at all the errors at the end of the log if there are several; esp. the initial exception. Also please capture all the callstacks (or attach the entire log) if opening a bug. Look at Ambari operation logs for HiveServer Interactive startup. If it has an error that is basically a timeout waiting for LLAP cluster, go to step 2 to determine why LLAP wasn’t able to start. This may also manifest as InvalidACL error (on an older HDP version). If the issue is after LLAP has started (i.e. in HiveServer2 startup) and there's no useful error message, see hiveServer2Interactive logs on the HiveServer2 Interactive service machine. Try to make sense of the error, and/or file a bug or contact someone in dev (Hive). Otherwise, try to make sense of the error, and/or file a bug or contact someone in dev (Ambari or Hive depending on the error). Check that the LLAP YARN app has enough capacity via ResourceManager UI, assuming it hasn't just failed (skip this step if the app is in failed state). If the app is neither running nor complete (i.e. it’s queued, accepted, etc.), capacity is missing in the cluster to start an app; check what takes up space on the cluster; if nothing looks unexpected, consult the sizing doc to ensure LLAP is set up properly. If the app is running (not failed), go to “Tracking URL: Application Master” link, and check if it can start enough LLAP daemons on the cluster (Desired vs Actual containers for LLAP component). If not, check if there’s enough capacity to start this number of containers – something else might be taking up the space; if nothing looks unexpected, consult the sizing doc to ensure LLAP is set up properly. If the app has failed, or there is enough capacity but the app doesn’t have enough containers, view the logs in the UI (or kill the app and download YARN logs). If the slider app status says “too many component failures” or a similar error, go directly to step 5; otherwise step 4. Check slider.log for errors (in the UI, it would be in the slider AM – usually the lowest numbered container). If the errors are container failures or there are no errors, go to step 5. Otherwise, try to make sense of the error, and/or file a bug or contact someone in dev (YARN or Hive depending on the error). If present, check daemon logs in one of the failed containers for errors. Start with (llap-daemon*.log, llap-daemon*.out). They would usually have the error at the end. Try to make sense of the error, and/or contact Hive team. If those are not present, check slider-agent.log instead; try to make sense of the error, and/or file a bug or contact someone in dev (YARN or Hive depending on the error). ... View more

This article provides an overview of various debugging aspects of the system when using LLAP - how are things run, where logs are located, what monitoring options are available, etc. See the parent article for architecture overview and specific issue resolution. HS2 when using LLAP When using LLAP (“Hive interactive”), Ambari starts a separate HS2 (HS2 Interactive). This HS2 is not connected to the other HS2, and has a separate connection string, machine (or ports), configuration and logs . HS2 interactive and LLAP configs in Ambari are split similarly to regular Hive configs, but have “-interactive-“ in their name. HS2 interactive logs are called “hiveServer2Interactive” and are separate from regular HS2 logs. YARN apps when using LLAP; getting query logs Ambari starts LLAP as a YARN app that is shared by everyone using the cluster. It should have N+1 containers, where N is the number of LLAP daemons, and there’s slider AM. LLAP YARN app is called “llap0” by default: Tez session YARN apps will always have one container (the query coordinator) when LLAP is used. In the below screenshot, LLAP cluster app and 2 single-container Tez sessions can be seen (this would be the case if 2 concurrent queries were configured in Ambari, see Cluster sizing). Because of the YARN deployment model, LLAP does not use jars/configs/etc. on local machine.They are packaged by Ambari at LLAP startup time, and are deployed in the container directories of the LLAP YARN app. In 2.6, the package with the jars and configs is rebuilt on every LLAP restart. Logs LLAP YARN app contains task logs for all the queries ; each LLAP writes a separate log file for every query. The name format of the log file containing the task logs for a single query on a single daemonis " <hive query Id>-<DAG name>.log(.done)", e.g. hive_20171110190337_9cd58b54-62e0-4940-b0b8-e8b62f877cbb-dag_1505982789269_1038_1.log.Every daemon would have one log file for each (or almost every) query. Over time, the log files for completed queries (with .done extension) are picked up by YARN log aggregation and removed from daemons' YARN directories. DAG name in the log file name corresponds to a Tez session app – e.g. in the above case, dag_1505982789269_ 1038_1 corresponds to query #1 in session represented by application_1505982789269_1038.This latter app (which is separate from the LLAP app) is the query coordinator (Tez AM) for the query and contains the Tez AM logs for it. Slider commands and views; LLAP app status overview Ambari uses Slider to launch LLAP. When debugging, it’s possible to use slider commands to affect the app without involving Ambari, e.g. “slider stop <name>” to stop the LLAP app by name (slider stop llap0). See “slider help” for more commands. Slider AM also exposes a status view, accessed from the YARN application page: The status view shows allocated and running containers, slider debugging info and also container logging directories on each machine, that can be used for manual debugging if there’s a problem with “yarn logs” command: LLAP daemon status and metrics On each machine where LLAP is running, it exposes a number of endpoints on port 15002 (by default), i.e. http://(machine name):15002/ The default view shows some metrics; other useful views are: /jmx – shows various useful metrics, discussed below /stacks - shows daemon stacks – threads that process query fragments are TezTR-*and IO-Elevator-Thread* /conf – shows LLAP configuration (the values actually in use) /iomem (only in 3.0) – shows debug information about cache contents and usage /peers – shows all LLAP nodes in the cluster /conflog (only in 3.0) – allows one to see and add/change logger and log level for this daemon without restart LLAP daemon metrics (JMX) At <host>:15002/jmx, one can see a lot of LLAP daemon metrics. Things to pay attention to: ExecutorsStatus contains the information about tasks queued and executing on LLAP. This is the most useful section when debugging stuck queries to see which tasks are running and in queue. ExecutorMemoryPerInstance, IoMemoryPerInstance, MaxJvmMemory contain the memory configuration in use (see the sizing section). java.class.path is the LLAP daemon classpath; for debugging class-loading issues. llap.daemon.log.dir is the LLAP daemon logging directory. LastGcInfo contains the information about last GC (more is available in the GC log in the logging directory). LlapDaemonIOMetrics, LlapDaemonCacheMetrics, LlapDaemonExecutorMetrics, LlapDaemonJvmMetrics contain LLAP performance metrics. The latter is especially useful when debugging limit issues. Tez view and performance debugging Tez view can be used to view Hive queries with LLAP, like the regular Hive queries; it has tabs with DAG representation, vertex swimlanes, etc. for each query. In particular, one can download debugging and performance data for the query. Grafana views LLAP exposes metrics to Ambari Timeline Server which can be viewed via Grafana Dashboards. Grafana dashboards can be viewed from Ambari via Hive -> Quick Links -> Hive Dashboard (Grafana) . This should open up Grafana dashboard for all HDP components. Click on “Home” dropdown to navigate to any of the 3 LLAP dashboards LLAP Daemon Dashboard snapshot that showing host level LLAP executor metrics. LLAP Overview Dashboard snapshot showing aggregated overview of memory from all nodes of LLAP cluster. LLAP Heatmaps Dashboard snapshot showing host level cache and executor heatmaps. There are several other metrics exposed via Grafana that can be useful for debugging. Full list of metrics and their description are available here https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-operations/content/grafana_hive_llap_dashboards.html ... View more

Introduction: how does LLAP fit into Hive LLAP is a set of persistent daemons that execute fragments of Hive queries. Query execution on LLAP is very similar to Hive without LLAP, except that worker tasks run inside LLAP daemons, and not in containers. High-level lifetime of a JDBC query: Without LLAP With LLAP Query arrives to HS2; it is parsed and compiled into “tasks” Query arrives to HS2; it is parsed and compiled into “tasks” Tasks are handed over to Tez AM (query coordinator) Tasks are handed over to Tez AM (query coordinator) Coordinator (AM) asks YARN for containers Coordinator (AM) locates LLAP instances via ZK Coordinator (AM) pushes task attempts into containers Coordinator (AM) pushes task attempts as fragment into LLAP RecordReader used to read data LLAP IO/cache used to read data or RecordReader used to read data Hive operators are used to process data Hive operators are used to process data* Final tasks write out results into HDFS Final tasks write out results into HDFS HS2 forwards rows to JDBC HS2 forwards rows to JDBC * sometimes, minor LLAP-specific optimizations are possible - e.g. sharing a hash table for map join Theoretically, a hybrid (LLAP+containers) mode is possible, but it doesn’t have advantages in most cases, so it’s rarely used (e.g.: Ambari doesn’t expose any knobs to enable this mode). In both cases, query uses a Tez session (YARN app with aTez AM serving as a query coordinator). In container case, AM will start more containers in the same YARN app; in LLAP case, LLAP itself runs as an external, shared YARN app, so Tez session will only have one container (the query coordinator). Inside LLAP daemon: execution LAP daemon runs work fragments using executors. Each daemon has a number of executors to run several fragments in parallel, and a local work queue. For the Hive case, fragments are similar to task attempts – mappers and reducers. Executors essentially “replace” containers – each is used by one task at a time; the sizing should be very similar for both. Inside LLAP daemon: IO Optionally, fragments may make use of LLAP cache and IO elevator (background IO threads). In HDP 2.6, it’s only supported for ORC format and isn’t supported for most ACID tables. In 3.0, support is added for text, Parquet, and ACID tables. In HDInsight, text format is also added in 2.6. Note that queries can still run in LLAP even if they cannot use the IO layer. Each fragment would only use one IO thread at a time. Cache stores metadata (on heap in 2.6, off heap in 3.0) and encoded data (off-heap); SSD cache option is also added in 3.0 (2.6 on HDInsight). ... View more

This is the central page that links to other LLAP debugging articles. You can find articles dealing with specific problems below; to provide some background and to help resolve other issues, there are also some helpful general-purpose articles: A one-page overview of LLAP architecture: https://community.hortonworks.com/articles/149894/llap-a-one-page-architecture-overview.html On the general debugging - where to find logs, UIs, metrics, etc.: https://community.hortonworks.com/articles/149896/llap-debugging-overview-logs-uis-etc.html You might also find this non-debugging LLAP sizing and setup guide interesting: https://community.hortonworks.com/articles/149486/llap-sizing-and-setup.html Before getting to specific issues, there are some limitations with LLAP that you should be aware of; the following significant features are not supported: HA (work in progress). doAs=true (no current plans to support it). Temporary functions (will not be supported; use permanent functions). CLI access (use beeline). Then, there are some articles about specific issues: LLAP doesn't start - https://community.hortonworks.com/articles/149899/investigating-when-llap-doesnt-start.html Queries are slow or stuck on LLAP - https://community.hortonworks.com/articles/149900/investigating-when-the-queries-on-llap-are-slow-or.html Investigating cache usage - https://community.hortonworks.com/articles/149901/investigating-llap-cache-hit-rate.html This list will be expanded in future with the issues that people are facing with LLAP. ... View more