@Michael Bronson For now we can enable "hive.server2.clear.dangling.scratchdir=true" for HiveServer2 via Custom hive-site. ... View more

@Michael Bronson I do not remember/think of any specific idem to check at this point, But as long as you are able to run your Hive Queries without any issue and HiveService checks are also running fine. I think we should be good. ... View more

@Michael Bronson You can add those scratchdir parameters inside the in custom hive-site.xml via Ambari UI by clicking the Add Property option. ... View more

@Michael Bronson To clean up the Hive scratch directory manually may not be a safe option for a multi-user environment (where multiple users might be executing the hive queries concurrently) since it will accidentally remove the scratch directory in use. ... View more

@Michael Bronson "hive.scratchdir.lock" : When true, holds a lock file in the scratch directory. If a Hive process dies and accidentally leaves a dangling scratchdir behind, the cleardanglingscratchdir tool will remove it. When false, does not create a lock file and therefore the cleardanglingscratchdir tool cannot remove any dangling scratch directories. Regarding your query "second is it safe to delete the folder - /tmp/hive/hive" >>> I do not think that we should do it on our own. As the whole purpose of the following JIRA was to introduce some tool like "cleardanglingscratchdir" to safely remove the scratch contents. https://issues.apache.org/jira/browse/HIVE-13429 . ... View more

@Michael Bronson As an immediate turnaround may be you would like to double / increase the value set for the following HDFS parameter "dfs.namenode.fs-limits.max-directory-items". It Defines the maximum number of items that a directory may contain. Cannot set the property to a value less than 1 or more than 6400000. (default value : 1048576) Go to Ambari -> HDFS -> Configs -> Advanced -> Custom hdfs-site and add the key (dfs.namenode.fs-limits.max-directory-item As per the apache hive docs there seems to be some parameters and tools available to deal with such issue. Although i have not personally tested those tools. But looks like they were introduced to deal with similar issue long back as part of https://issues.apache.org/jira/browse/HIVE-13429 For example i see that the Hive Config "hive.exec.scratchdir" points to the "/tmp/hive" dir. Can you please check and let us know what is the value set for the following parameter "hive.scratchdir.lock". (if not set then default value will be "false"? Additionally you might want to refer about "hive.server2.clear.dangling.scratchdir" and "hive.start.cleanup.scratchdir" parameters of Hive Server config. Please refer to [1] the following link to know more about those parameters. There is a tool "cleardanglingscratchdir" mentioned as part of the link [2] may be you would like to read more about it. # hive --service cleardanglingscratchdir [-r] [-v] [-s scratchdir]     -r      dry-run mode, which produces a list on console     -v      verbose mode, which prints extra debugging information     -s      if you are using non-standard scratch directory . [1] https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.scratchdir.lock. [2] https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2#SettingUpHiveServer2-ClearDanglingScratchDirTool ... View more

@Adil BAKKOURI Your Zookeeper process command output shows the log name something like following: root@RHBigData1:~# ps -ef | grep -i zookeeper . . -Dzookeeper.log.file=zookeeper-zookeeper-server-RHBigData1.log . I doubt that your Zookeeper Hostname might not be correct. Usually the zookeeper log file name is generated based on the FQDN of the node. So if your Zookeeper FQDN is correctly set then it should be showing something like following: # hostname -f master.rh.bigdata.cluster . So please verify of your Zookeeper host has the FQDN setup correctly and then restart the Zookeeper after fixing the FQDN. Later you should see the zookeeper log file name something like "zookeeper-zookeeper-server-master.rh.bigdata.cluster.log" ... View more

@Adil BAKKOURI Ambari Simply makes use of the Zookeeper Smoke Test "zkSmoke.sh" script to verify the Zookeeper Connection and if it can create & delete the dummy ZNode "/zk_smoketest" inside the zookeeper host or not? As you see the following message: ExecutionFailed: Execution of '/var/lib/ambari-agent/tmp/zkSmoke.sh /usr/hdp/current/zookeeper-client/bin/zkCli.sh ambari-qa /usr/hdp/current/zookeeper-client/conf 2181 False kinit no_keytab no_principal /var/lib/ambari-agent/tmp/zkSmoke.out' returned 4. zk_node1=master.rh.bigdata.cluster  . . Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /zk_smoketest   Which means your Zookeeper process might not be running file. Most probably the Zookeeper port might not be accessible on host "master.rh.bigdata.cluster" So you will need to first check if your Zookeeper is running fine or not on host "master.rh.bigdata.cluster" and if the Zookeeper port 2181 is listening fine or not? # ps -ef | grep -i zookeeper # netstat -tnlpa | grep 2181 # service iptables stop Most Possible Cause: If you see that Zookeeper process is not running then please try to restart it and check the Zookeper Logs to find any issue/ startup failure and please share the log. Most probably once your Zookeeper is running fine and the port 2181 is accessible then your Zookeeper Check Should also run fine. Firewall or Port Blocking issue: So please go to the host where the SmokeTest is failing and then try to telnet to Zookeeper Host & Port to see if that is accessible? # telnet master.rh.bigdata.cluster  2181 (OR) # nc -v master.rh.bigdata.cluster  2181 Manual Test: You your self can try running the same command to verify this Zookeeper Zmoke Test run Try running this command on different Zookeeper nodes as it is. # /var/lib/ambari-agent/tmp/zkSmoke.sh /usr/hdp/current/zookeeper-client/bin/zkCli.sh ambari-qa /usr/hdp/current/zookeeper-client/conf 2181 False kinit no_keytab no_principal /var/lib/ambari-agent/tmp/zkSmoke.out . . ... View more