question Re: HDFS is not accessible from an user after kerberos implementation in Support Questions

question Re: HDFS is not accessible from an user after kerberos implementation in Support Questions https://community.cloudera.com/t5/Support-Questions/HDFS-is-not-accessible-from-an-user-after-kerberos/m-p/268878#M206457 <a href="https://community.cloudera.com/t5/user/viewprofilepage/user-id/49879">@pritam_konar</a>  In reality a user shouldn't be able to execute or kinit with the hdfs keytab but have a keytab created for the specific user and when need be deleted when the user is disabled on the cluster typically this user setup happens on the edge node where the hadoop client software are installed and is the recommended setup for giving users access to the cluster.Below is a demo of the user konar when he attempts to access services in a kerberized cluster# su - konar[konar@simba ~]$ id uid=1024(konar) gid=1024(konar) groups=1024(konar)Now try to list the directories in HDFS[konar@simba ~]$ hdfs dfs -ls /Error  19/08/24 23:59:25 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "simba.kenya.ke/192.168.0.87"; destination host is: "simba.kenya.ke":8020; Below is the desired output when the user konar attempts to use the hdfs headless keytab,[konar@simba ~]$ kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs-jair@KENYA.KE kinit: Permission denied while getting initial credentials To enable a user to access the cluster, on the Kerberos server as the root (Keberos admin)  do the following steps,Assumption  Realm is KENYA.KE  and KDC  host is simba and you have root access on the KDC.Create the admin principle for user konar[root@simba ~]# kadmin.local Authenticating as principal root/admin@KENYA.KE with password. kadmin.local: addprinc konar@KENYA.KE WARNING: no policy specified for konar@KENYA.KE; defaulting to no policy Enter password for principal "konar@KENYA.KE": Re-enter password for principal "konar@KENYA.KE": Principal "konar@KENYA.KE" created. kadmin.local: qValidate the principal was created using the subcommand listprincs [List principals] and limiting the output by restricting to  konar classic Unix stuff[root@simba ~]# kadmin.local Authenticating as principal root/admin@KENYA.KE with password. kadmin.local: listprincs *konar konar@KENYA.KEType q [quit] to exit the kadmin utility Generate the keytab Generate keytab for user konar using the ktutil, it's good to change to /tmp or whatever you choose so you know the location of the generated keytab your encryption Algorithm could be different but this should work[root@simba tmp]# ktutil ktutil: addent -password -p konar@KENYA.KE -k 1 -e RC4-HMAC Password for konar@KENYA.KE: ktutil: wkt konar.keytab ktutil: q Validate the keytab creationCheck the keytab was generated in the current directory, notice the file permissions!![root@simba tmp]# ls -lrt -rw------- 1 root root 58 Aug 25 18:22 konar.keytab As root copy the generate keytab to the home directory of user konar typically on the edge node[root@simba tmp]# cp konar.keytab /home/konar/ Change to konar's home dir and vaildate the copy was successful [root@simba tmp]# cd /home/konar/ [root@simba konar]# ll total 4 -rw------- 1 root root 58 Aug 25 18:28 konar.keytab Change file ownershipChange the file permission on the konar.keytab so that user konar has the appropriate permissions.[root@simba konar]# chown konar:konar konar.keytab [root@simba konar]# ll total 4 -rw------- 1 konar konar 58 Aug 25 18:28 konar.keytab Switch to user konar and validate that the user has can't  still access to hdfs$ hdfs dfs -ls /[konar@simba ~]$ hdfs dfs -ls /Output 19/08/25 18:36:44 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "simba.kenya.ke/192.168.0.87"; destination host is: "simba.kenya.ke":8020; The kerberos klist also confirms that[konar@simba ~]$ klist klist: No credentials cache found (filename: /tmp/krb5cc_1024) As user Konar now try to kinit with the correct principal, the first step is to identify the correct principal[konar@simba ~]$ klist -kt konar.keytab Keytab name: FILE:konar.keytab KVNO Timestamp Principal ---- ------------------- ------------------------------------------------------ 1 08/25/2019 18:22:34 konar@KENYA.KEThe above shows the konar user keytab is valid with the principal in the output Now user konar can grab a valid ticket ûsing the below snippet concatenating the keytab + principal [konar@simba ~]$ kinit -kt konar.keytab konar@KENYA.KEThe above should throw any error Now validate the user has a valid ticket[konar@simba ~]$ klist Ticket cache: FILE:/tmp/krb5cc_1024 Default principal: konar@KENYA.KEValid starting Expires Service principal 08/25/2019 18:53:40 08/26/2019 18:53:40 krbtgt/KENYA.KE@KENYA.KEBravo you have a valid ticket and hence access to the cluster let's validate that the below  HDFS list  directory should succeed [konar@simba ~]$ hdfs dfs -ls / Found 10 items drwxrwxrwx - yarn hadoop 0 2018-12-17 21:53 /app-logs drwxr-xr-x - hdfs hdfs 0 2018-09-24 00:22 /apps drwxr-xr-x - yarn hadoop 0 2018-09-24 00:12 /ats drwxr-xr-x - hdfs hdfs 0 2018-09-24 00:12 /hdp drwxr-xr-x - mapred hdfs 0 2018-09-24 00:12 /mapred drwxrwxrwx - mapred hadoop 0 2018-09-24 00:12 /mr-history drwxr-xr-x - hdfs hdfs 0 2018-12-17 19:16 /ranger drwxrwxrwx - spark hadoop 0 2019-08-25 18:59 /spark2-history drwxrwxrwx - hdfs hdfs 0 2018-10-11 11:16 /tmp drwxr-xr-x - hdfs hdfs 0 2018-09-24 00:23 /user User konar can now list and execute jobs on the cluster !!!! as reiterated the konar user in a recommended architecture should be on the edge node.    Sun, 25 Aug 2019 17:28:06 GMT Shelton 2019-08-25T17:28:06Z