I have just enabled kerberos on the hadoop cluster. The whole process went fairly smooth. However after I needed to restart all the services I noticed that it over 30 min for the NameNode to start up. During these 30 min it seems that hdfs did not have a valid TGT based on the messages below. After patiently waiting and thinking it is going to fail any moment it in fact DID come up. My question is why it took so long and what was the problem of obtaining TGT from the very beginning? 2018-10-27 23:28:54,899 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://*******:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 18/10/27 23:28:54 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] safemode: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "*******/11*.11*.11*.11*"; destination host is: "***.***.***":8020; 18/10/27 23:29:09 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] safemode: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "*******/11*.11*.11*.11*"; destination host is: "************":8020; ... View more

I'm trying to start the metrics collector , but getting a strange error instead: resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/sbin/ambari-metrics-collector --config /etc/ambari-metrics-collector/conf start' returned 1. Tue Oct 23 18:05:37 EDT 2018 Starting HBase. starting master, logging to /disks/disk1/log/ambari-metrics-collector/hbase-ams-master-node4.hdp.com.out /usr/lib/ams-hbase/bin/hbase-daemon.sh: line 189: /disks/disk1/log/ambari-metrics-collector/hbase-ams-master-node4.hdp.com.out: No space left on device head: cannot open ‘/disks/disk1/log/ambari-metrics-collector/hbase-ams-master-node4.hdp.com.out’ for reading: No such file or directory /usr/sbin/ambari-metrics-collector: line 81: /disks/disk1/run/ambari-metrics-collector/ambari-metrics-collector.pid: No space left on device ERROR: Cannot write pid /disks/disk1/run/ambari-metrics-collector/ambari-metrics-collector.pid. it is complaining there is no space on device. /disks/disk1/log/ambari-metrics-collector/ambari-metrics-collector.out shows the same thing: Java HotSpot(TM) 64-Bit Server VM warning: Cannot open file /disks/disk1/log/ambari-metrics-collector/collector-gc.log-201810231817 due to No space left on device log4j:ERROR setFile(null,true) call failed. java.io.FileNotFoundException: /disks/disk1/log/ambari-metrics-collector/ambari-metrics-collector.log (No space left on device) ...... but # df -h /disks/disk1/ Filesystem Size Used Avail Use% Mounted on /dev/sdb1 4.9G 760M 4.2G 16% /disks/disk1 There is clearly some space there. How much space is really needed to write the output file? Thanks, Alex ... View more