How do I use LLAP SSD cache？

[#1]

I test LLAP cache using SSD. I set hive.llap.io.allocator.mmap to true, and set hive.llap.io.allocator.mmap.path to SSD mount directory '/data1'. Single LLAP daemon memory is about 59GB and Xmx is about 57GB, and cache is about 48GB.

Then LLAP is started successfully and local directory '/data1/llap-6454532602074638940' is created. As Shown below:

I did TPC-DS test with 200GB Text data. I observe the Cache Metrics by 15002 port as well as Grafana. As Shown below:

We could see about 41 GB cache was generated on single LLAP daemon after several TPC-DS SQL queries. However，I found no data produced in local SSD cache directory '/data1/llap-6454532602074638940', and in other words, this directory was always empty.

I also observed the memory usage of physical machine when llap was running. As shown below:

As well as the memory usage of physical machine when llap was turned off. As shown below:

From these two pictures, we can see it seems that single llap daemon hold about 80GB memory(59GB llap daemon memory was configured in fact) as well as 20GB operating system cache.

So here's the problem:

1) How does LLAP use SSD caching? Why is there no data in SSD local mount directory?

2)Why does SSD use more memory than it actually sets up? Is the data still cached in memory instead of SSD?

3)How to use LLAP SSD cache correctly? How do I monitor llap cache usage?

Thanks very much!

[#2]

I am meeting the same issue, is there any update?

and I can't find an official document telling details about enabling LLAP SSD caching, if anyone knows where to find the document please share here, thanks a lot!

[#3]

I'm also having same issue. I'm using HDP3.1.0 and enable it with Ambari like,

I did some trials with changing cache size for LLAP daemon, heap size for LLAP daemon, etc. but no luck at all. I can't see any OS files under the OS path "/hadoop/hive/llap".

BTW, when I run Hive query with LLAP, I can see some usage of LLAP daemon cache after turning on "Turn SSD cache On" with no usage of the OS path "hadoop/hive/llap".

Does it mean, LLAP uses OS memory as LLAP cache as well as SSD?

[#4]

After investigation and testing, I found that actually Hive LLAP daemon seems to use the SSD device with the OS path (in my case "OS path "/hadoop/hive/llap") even when there is no OS file under the directory.

If you are using Linux, you can see difference of usage from df command and usage from du command.

In my case, the difference of them was matched to the cache usage from LLAP daemon UI.

Also, you can confirm there is no evicted with "http://XXXX:15002/iomem" like,

ORC cache summary: 0 locked, 556529 unlocked, 0 evicted, 0 being moved,18236342272total used space

I think, this is a kind of direct access to the block device (not via OS FS.)

[#5]

https://issues.apache.org/jira/secure/attachment/12805509/HIVE-13029.4.patch

I found the comment of source code.

+  private ByteBuffer preallocate(int arenaSize) {
+    if (isMapped) {
+      Preconditions.checkArgument(isDirect, "All memory mapped allocations have to be direct buffers");
+      try {
+        File rf = File.createTempFile("arena-", ".cache", cacheDir.toFile());
+        RandomAccessFile rwf = new RandomAccessFile(rf, "rw");
+        rwf.setLength(arenaSize); // truncate (TODO: posix_fallocate?)
+        ByteBuffer rwbuf = rwf.getChannel().map(MapMode.PRIVATE, 0, arenaSize);
+        // A mapping, once established, is not dependent upon the file channel that was used to
+        // create it. delete file and hold onto the map
+        rwf.close();
+        rf.delete();
+        return rwbuf;
+      } catch (IOException ioe) {
+        LlapIoImpl.LOG.warn("Failed trying to allocate memory mapped arena", ioe);
+        // fail similarly when memory allocations fail
+        throw new OutOfMemoryError("Failed trying to allocate memory mapped arena: " + ioe.getMessage());
+      }
+    }
+    return isDirect ? ByteBuffer.allocateDirect(arenaSize) : ByteBuffer.allocate(arenaSize);
+  }

So, llap daemon creates tmp file and delete the file immediately, but keep using it.

This should be the reason of difference of output from df command and du command.

a little bit trickey, isnt it?

[#6]

@tomo_hirano 

Understood, it makes sense.

I also confirmed the disk usage kept increasing while there was no file created.

Thanks for sharing!