@Sampath Kumar, @SHAIKH FAIROZ AHMED, @Jack Marquez, did you find the solution to this problem? I am facing it too. ... View more

What I did was editing the code giving '/tmp/hive/hive' as the root directory. I should have given '/tmp' at the first place. ... View more

Good idea. Felt bad and lazy for not inspecting the code before. Got again OOM however. Thank you anyways. I will try to spare some time to take a look at the code. I will update the question if I can progress. ... View more

Initial solution i tried was a script in below answer https://community.hortonworks.com/answers/92110/view.html I modified this script to work in /tmp/hive/hive path but it received OOM error. ... View more

Follow up... ... View more

@Geoffrey Shelton Okot Unfortunately, I couldn't start HDFS services this way, neither. Thank you very much though. ... View more

Hi @Geoffrey Shelton Okot, I had tried hadoop namenode -format before but tried again and received the same exception: 17/12/05 09:46:25 ERROR namenode.NameNode: Failed to start namenode. org.apache.hadoop.hdfs.qjournal.client.QuorumException: Could not format one or more JournalNodes. 2 exceptions thrown: 10.0.109.11:8485: Directory /hadoop/hdfs/journal/testnamespace is in an inconsistent state: Can't format the storage directory because the current directory is not empty. at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.checkEmptyCurrent(Storage.java:482) at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:558) at org.apache.hadoop.hdfs.qjournal.server.JNStorage.format(JNStorage.java:185) at org.apache.hadoop.hdfs.qjournal.server.Journal.format(Journal.java:217) at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.format(JournalNodeRpcServer.java:145) at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.format(QJournalProtocolServerSideTranslatorPB.java:145) at org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:25419) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347) This time additionally I deleted content of /hadoop/hdfs/journal/testnamespace but nothing changed. Command ended up with the same exception. ... View more

So.. What are the steps for reinstall? Is there any way to start from only HDP installation but keeping OS level changes as prerequisite and also ambari installation? Does command ambari-server reset work for that? ... View more

If recovery steps will take more than re-install and/or give me an unstable cluster then its better to reinstall. What I anticipate from your answer you mean such costs, right? ... View more

For a complete YARN administration I would recommend below famous book by Sam R. Alapati: https://www.amazon.com/Expert-Hadoop-Administration-Addison-Wesley-Analytics/dp/0134597192 Chapter 13 describes how YARN works and Chapter 18 explains how you can make tuning. Below page describes not only Hive parameters for Hive tuning but also mentions what to do on Yarn. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_hive-performance-tuning/content/ch_hive-perf-tuning-intro.html I would suggest reading this to understand how capacity scheduler works. This way you understand how it works and what to do to share resources: https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html ... View more

We have just overcame such issue. I assume you have enabled pre-emption on YARN. After that, a little bit of Hive tuning is kind of prerequisite for all users when enabling LLAP (interactive query component). Most common problem with starting interactive server is conflicts between YARN and Hive for resource usage. There are some main parameters which are mostly effective on starting (and running stable) LLAP: hive.llap.daemon.yarn.container.mb: Suggestion on Ambari is pretty clear: Total memory used by individual LLAP daemons (YARN Container size). This includes memory for the cache as well as for the query execution. Should be larger than the sum of the Daemon cache size and the daemon heap size, and should leave some headroom after this (In most cases: cache size + heap size + headroom = Memory Per Daemon) We also tried to keep it slightly below yarn.scheduler.maximum-allocation-mb value of yarn config. This way we prevent resource conflict with YARN in basic level. llap_headroom_space : Maximum headroom reserved from the YARN container running LLAP daemons. This is an upper limit used during automatic size calculations, and the actual value may be lower. We left this as recommended value which is very low compared to others. It is about %6 of llap_heap_size llap_heap_size: LLAP Daemon Heap Size in MB. hive.llap.io.memory.size: The amount of memory reserved for Hive's optimized in-memory cache. In memory cache size should be %20 of llap heap size And below parameters are number of executors which are set in default according to number of CPU's num_llap_nodes_per_llap_daemons: Number of Node(s) on which Hive LLAP daemon runs.<br> We set this value to number of our nodes (nodemanagers / datanodes). hive.llap.daemon.num.executors: The number of fragments that a single LLAP daemon will run concurrently. Usually, this will be the same as the number of available CPUs<br> We set this to %40 of number of CPUs hive.llap.io.threadpool.size: Specify the number of threads to use for low-level IO thread pool.<br> We also set this one with the same value of number of executors. After updating these values, you can save configs and start interactive server. Afterwards, you may need to update queue capacities on Yarn. If you are using Ambari, Yarn Queue Manager view is ideal as it provides ability to view and adjust the values visually. Here you should check if LLAP uses more resource over default queue. If you don't have YARN Queue Manager, you can adjust below parameters and variants of the for default queue. yarn.scheduler.capacity.root.llap.capacity=12.0 yarn.scheduler.capacity.root.llap.maximum-am-resource-percent=1 yarn.scheduler.capacity.root.llap.maximum-capacity=12.0 ... View more

If it is only one datanode failing but not all of them, it may be failing due to a volume failure. You can check Namenode web-ui to see if you are facing any volume failures: http://<active-namenode-host>:50070/dfshealth.html#tab-datanode-volume-failures or just http://<active-namenode-host>:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystemState For plain JMX data to see whether or not you're having volume failures (it is not physical disk error all the time. It may occur after logical write issues). ... View more