https://issues.apache.org/jira/secure/attachment/12805509/HIVE-13029.4.patch   I found the comment of source code. + private ByteBuffer preallocate(int arenaSize) { + if (isMapped) { + Preconditions.checkArgument(isDirect, "All memory mapped allocations have to be direct buffers"); + try { + File rf = File.createTempFile("arena-", ".cache", cacheDir.toFile()); + RandomAccessFile rwf = new RandomAccessFile(rf, "rw"); + rwf.setLength(arenaSize); // truncate (TODO: posix_fallocate?) + ByteBuffer rwbuf = rwf.getChannel().map(MapMode.PRIVATE, 0, arenaSize); + // A mapping, once established, is not dependent upon the file channel that was used to + // create it. delete file and hold onto the map + rwf.close(); + rf.delete(); + return rwbuf; + } catch (IOException ioe) { + LlapIoImpl.LOG.warn("Failed trying to allocate memory mapped arena", ioe); + // fail similarly when memory allocations fail + throw new OutOfMemoryError("Failed trying to allocate memory mapped arena: " + ioe.getMessage()); + } + } + return isDirect ? ByteBuffer.allocateDirect(arenaSize) : ByteBuffer.allocate(arenaSize); + } So, llap daemon creates tmp file and delete the file immediately, but keep using it. This should be the reason of difference of output from df command and du command. a little bit trickey, isnt it? ... View more

After investigation and testing, I found that actually Hive LLAP daemon seems to use the SSD device with the OS path (in my case " OS path "/hadoop/hive/llap") even when there is no OS file under the directory. If you are using Linux, you can see difference of usage from df command and usage from du command. In my case, the difference of them was matched to the cache usage from LLAP daemon UI. Also, you can confirm there is no evicted with "http://XXXX:15002/iomem" like, ORC cache summary: 0 locked, 556529 unlocked, 0 evicted, 0 being moved,18236342272total used space I think, this is a kind of direct access to the block device (not via OS FS.) ... View more

I'm also having same issue. I'm using HDP3.1.0 and enable it with Ambari like,   I did some trials with changing cache size for LLAP daemon, heap size for LLAP daemon, etc. but no luck at all. I can't see any OS files under the OS path "/hadoop/hive/llap".   BTW, when I run Hive query with LLAP, I can see some usage of LLAP daemon cache after turning on "Turn SSD cache On" with no usage of the OS path "hadoop/hive/llap". Does it mean, LLAP uses OS memory as LLAP cache as well as SSD? ... View more

I had same issue, and we are using HDP 3.1.0.0-78 . https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/patch_tez.html TEZ-3894 seems to be already applied to HDP 3.1. (Also, I've checked the source code a little, yes, it looks already applied.) But I still have this issue... I can avoid this issue by changing fs.permissions.umask-mode from "077" to "022" in a HS2 session. 0: jdbc:hive2://XXXX > set fs.permissions.umask-mode=022; So I guess, this issue may not be fixed completely with TEZ-3894 (with HDP 3.1.0.0-78)... ... View more

Looks, it's tez issue comes from "fs.permissions.umask-mode" setting. https://community.hortonworks.com/questions/246302/hive-tez-vertex-failed-error-during-reduce-phase-h.html ... View more

I'm having same issue with HDP3.1 (Tez 0.9.1). I can reproduce it with: 1) create two files - file1.csv and file2.csv 2) add two fields to the csv files as below one,two one,two one,two 3) create external table use testdb; create external table test1(s1 string, s2 string) row format delimited fields terminated by ',' stored as textfile location '/user/usera/test1'; 4) Copy one csv file to hdfs - /user/usera/test1 hdfs dfs -put ./file1.csv /user/usera/test1/ 5) select count(*) from testdb.test1; => works fine. 6) copy the second csv file to HDFS hdfs dfs -put ./file2.csv /user/usera/test1/ 7) select * from testdb.test1; => Can see the data in both hdfs files. 8) select count(*) form testdb.test1; => Get this problem. And we can see following error in the mapper task's log. 2019-05-17 10:08:10,317 [INFO] [Fetcher_B {Map_1} #1] |shuffle.Fetcher|: Failed to read data to memory for InputAttemptIdentifier [inputIdentifier=1, attemptNumber=0, pathComponent=attempt_1557383221332_0289_1_00_000001_0_10003, spillType=0, spillId=-1]. len=25, decomp=11. ExceptionMessage=Not a valid ifile header 2019-05-17 10:08:10,317 [WARN] [Fetcher_B {Map_1} #1] |shuffle.Fetcher|: Failed to shuffle output of InputAttemptIdentifier [inputIdentifier=1, attemptNumber=0, pathComponent=attempt_1557383221332_0289_1_00_000001_0_10003, spillType=0, spillId=-1] from XXXXX java.io.IOException: Not a valid ifile header at org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.verifyHeaderMagic(IFile.java:859) at org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.isCompressedFlagEnabled(IFile.java:866) at org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.readToMemory(IFile.java:616) at org.apache.tez.runtime.library.common.shuffle.ShuffleUtils.shuffleToMemory(ShuffleUtils.java:121) at org.apache.tez.runtime.library.common.shuffle.Fetcher.fetchInputs(Fetcher.java:950) at org.apache.tez.runtime.library.common.shuffle.Fetcher.doHttpFetch(Fetcher.java:599) at org.apache.tez.runtime.library.common.shuffle.Fetcher.doHttpFetch(Fetcher.java:486) at org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:284) at org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:76) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) I think, it's similar to https://issues.apache.org/jira/browse/TEZ-3699 I've confirmed the patch already applied to tez with HDP 3.1. So I guess, it's new bug with Tez 0.9.x (I confirmed there is no problem with HDP2.6/Tez 0.7.0). Any idea? ... View more

One update. Hive seems to have an issue about handling "viewfs://"? For example, hive db creation got failed with permission error even if the permission of the target hdfs dir is 777. HDFS permission: hdfs dfs -ls -d viewfs://fed/user/hadoop/warehouse/hadoop_viewfs6.db drwxrwxrwx - hadoop hadoop 0 2019-04-24 07:44 viewfs://fed/user/hadoop/warehouse/hadoop_viewfs6.db Hive DB creation got failed with the hdfs dir. 0: jdbc:hive2://XXXX > create database hadoop_viewfs6 location '/user/hadoop/warehouse/hadoop_viewfs6.db'; Error: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [hadoop] does not have [WRITE] privilege on [viewfs://fed/user/hadoop/warehouse/hadoop_viewfs6.db] (state=42000,code=40000 Succeeded with "hdfs://" 0: jdbc:hive2://XXXX > create database hadoop_viewfs6 location 'hdfs://ns1/user/hadoop/warehouse/hadoop_viewfs6.db'; ... View more

I had same issue and solved by changing "yarn.nodemanager.resource.memory-mb" from 468GB to 200GB with Amabri (There were following 11 changes.) (I shared it because it's really hard to find out the reason by reading error messages...) ... View more

Hi, I'm facing completely same issue with HDP2.6.2. HDFS client have to wait about 20 seconds when the 1st NameNode is powered off. (Actually, we had this issue then the 1st NameNode had kernel hang (kernel panic).) Do you find good solution or workaround??? If so, please share it. Any information will help us ! ... View more