I test LLAP cache using SSD. I set hive.llap.io.allocator.mmap to true, and set hive.llap.io.allocator.mmap.path to SSD mount directory '/data1'. Single LLAP daemon memory is about 59GB and Xmx is about 57GB, and cache is about 48GB. Then LLAP is started successfully and local directory '/data1/llap-6454532602074638940' is created. As Shown below: I did TPC-DS test with 200GB Text data. I observe the Cache Metrics by 15002 port as well as Grafana. As Shown below: We could see about 41 GB cache was generated on single LLAP daemon after several TPC-DS SQL queries. However，I found no data produced in local SSD cache directory '/data1/llap-6454532602074638940', and in other words, this directory was always empty. I also observed the memory usage of physical machine when llap was running. As shown below: As well as the memory usage of physical machine when llap was turned off. As shown below: From these two pictures, we can see it seems that single llap daemon hold about 80GB memory(59GB llap daemon memory was configured in fact) as well as 20GB operating system cache. So here's the problem: 1) How does LLAP use SSD caching? Why is there no data in SSD local mount directory? 2)Why does SSD use more memory than it actually sets up? Is the data still cached in memory instead of SSD? 3)How to use LLAP SSD cache correctly? How do I monitor llap cache usage? Thanks very much!
... View more