I have two realms in my environment.
for ex. forest.org - currently all users are mapped to this domain ( Redhat IDM)
data.forest.org - Local MIT KDC (hadoop principlas)
Currently firstname.lastname@example.org is facing issues while accessing hdfs. Please find the error
19/10/17 03:43:38 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1571298216320
Kindly help us to fix this issue.
As this includes cross realm authentication. You can set below env variable to identify what part of the kerberos auth is not complete.
#export HADOOP_OPTS=" -Dsun.security.krb5.debug=true"
#hdfs dfs -ls /
Below are the step by steup auth steps with cross realm, If hostname to realm is defined correct in krb5.conf, where NN FQDN should resolve to Hadoop realm.
This should log some additional debug, first log should show getting krbtgt/FOREST.COM@FOREST.COM using cname <user>@FOREST.COM
Further it should show the debug about getting cross-realm tgt and this is based on CAPATH (if there are any intermediate trust between FOREST.COM and hadoop MIT kdc) java will try various combinations of trust, starting from krbtgt/FOREST.COM@HADOOP.REALM. If you get this tgt and dont see any message like "Server not found in kerberos database" then last step in auth process would be successful based on if HADOOP.REALM is able to decrypt the trust tgt. This part relies on availability of the this tgt on hadoop MIT and the password set for this principal(same password set in IDM while creating trust must be set on the HADOOP realm KDC for this last step to be complete).
Once this initial krbtgt process is successful, client will request service principal for nn/<FQDN>@HADOOP.REALM and completes further authentication cycle.
Most of the troubleshooting part will include the trust setup between IDM and MIT (creating krbtgt cross realm principals etc..) and encryption types used for these krbtgt principals.
krb5kdc.log (on IDM and MIT) will give good details as well about tgt/tgs requests and failures.