Demystifying Delegation Token

[#1]

The concept of Delegation token is introduced to avoid frequent authentication check against Kerberos(AD/MIT). After the initial authentication against Namenode using Keberos, any subsequent authentication can be done without Kerberos service ticket(ot TGT). Once the client authentication with Kerberos for Namenode is successfull, The client can get a delegation token from Namenode.This token has expiration and max issue date. But this can be renewed up to max issue date. In this article, We are going to see how delegation token can be created with the initial authentication and even if you delete initial TGT, You can still list the content with the help of delegation token

1). List the ticket from Kerberos.

root@hdptest-1 ~]# cd /etc/security/keytabs/

[root@hdptest-1 keytabs]# klist -kt hdfs.headless.keytab

Keytab name: FILE:hdfs.headless.keytab

KVNO Timestamp Principal

---- ----------------- --------------------------------------------------------

0 07/23/16 02:44:25 hdfs-hdptest@LAB.HORTONWORKS.NET

0 07/23/16 02:44:25 hdfs-hdptest@LAB.HORTONWORKS.NET

0 07/23/16 02:44:25 hdfs-hdptest@LAB.HORTONWORKS.NET

0 07/23/16 02:44:25 hdfs-hdptest@LAB.HORTONWORKS.NET

0 07/23/16 02:44:25 hdfs-hdptest@LAB.HORTONWORKS.NET

2).Perform Kinit.

[root@hdptest-1 keytabs]# kinit -kt hdfs.headless.keytab hdfs-hdptest@LAB.HORTONWORKS.NET

[root@hdptest-1 keytabs]# klist

Ticket cache: FILE:/tmp/krb5cc_0

Default principal: hdfs-hdptest@LAB.HORTONWORKS.NET

Valid starting Expires Service principal

08/04/16 22:18:50 08/05/16 08:18:50 krbtgt/LAB.HORTONWORKS.NET@LAB.HORTONWORKS.NET

renew until 08/11/16 22:18:50

3). List HDFS contents using

[root@hdptest-1 keytabs]# hadoop fs -ls /

Found 11 items

drwxrwxrwx - yarn hadoop 0 2016-08-03 05:51 /app-logs

drwxr-xr-x - hdfs hdfs 0 2016-08-03 05:53 /apps

drwxr-xr-x - yarn hadoop 0 2016-07-23 00:16 /ats

drwxr-xr-x - hdfs hdfs 0 2016-07-23 00:16 /hdp

drwxr-xr-x - mapred hdfs 0 2016-07-23 00:16 /mapred

drwxrwxrwx - mapred hadoop 0 2016-07-23 00:16 /mr-history

drwxr-xr-x - hdfs hdfs 0 2016-07-25 05:25 /ranger

drwxrwxrwx - hdfs hdfs 0 2016-07-23 02:51 /tmp

drwxr-xr-x - hdfs hdfs 0 2016-08-03 05:50 /user

drwxr-xr-x - hadoopadmin hdfs 0 2016-08-03 05:52 /zone_encr

drwxr-xr-x - hadoopadmin hdfs 0 2016-08-03 05:49 /zone_encr2

4). Generate the Delegation token, This is based on the existing ticket you have.

[root@hdptest-1 keytabs]# hdfs fetchdt --renewer hdfs my.delegation.token

16/08/04 22:19:44 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 15 for hdfs on 172.26.67.6:8020

Fetched token for 172.26.67.6:8020 into file:/etc/security/keytabs/my.delegation.token

Fetched token for 172.26.67.8:9292 into file:/etc/security/keytabs/my.delegation.token

[root@hdptest-1 keytabs]# klist

Ticket cache: FILE:/tmp/krb5cc_0

Default principal: hdfs-hdptest@LAB.HORTONWORKS.NET

Valid starting Expires Service principal

08/04/16 22:18:50 08/05/16 08:18:50 krbtgt/LAB.HORTONWORKS.NET@LAB.HORTONWORKS.NET

renew until 08/11/16 22:18:50

5). Destroy the ticket cache.

[root@hdptest-1 keytabs]# kdestroy

[root@hdptest-1 keytabs]# export HADOOP_TOKEN_FILE_LOCATION=/etc/security/keytabs/my.delegation.token

[root@hdptest-1 keytabs]# klist

klist: No credentials cache found (ticket cache FILE:/tmp/krb5cc_0)

6). List the HDFS content.Even though you dont have the ticket you are still able to do the listing thanks to Delegation token.

[root@hdptest-1 keytabs]# hadoop fs -ls /

Found 11 items

drwxrwxrwx - yarn hadoop 0 2016-08-03 05:51 /app-logs

drwxr-xr-x - hdfs hdfs 0 2016-08-03 05:53 /apps

drwxr-xr-x - yarn hadoop 0 2016-07-23 00:16 /ats

drwxr-xr-x - hdfs hdfs 0 2016-07-23 00:16 /hdp

drwxr-xr-x - mapred hdfs 0 2016-07-23 00:16 /mapred

drwxrwxrwx - mapred hadoop 0 2016-07-23 00:16 /mr-history

drwxr-xr-x - hdfs hdfs 0 2016-07-25 05:25 /ranger

drwxrwxrwx - hdfs hdfs 0 2016-07-23 02:51 /tmp

drwxr-xr-x - hdfs hdfs 0 2016-08-03 05:50 /user

drwxr-xr-x - hadoopadmin hdfs 0 2016-08-03 05:52 /zone_encr

drwxr-xr-x - hadoopadmin hdfs 0 2016-08-03 05:49 /zone_encr2

7). Check if we can get the Delegation token without the initial ticket from kerberos.

[root@hdptest-1 keytabs]# unset HADOOP_TOKEN_FILE_LOCATION

[root@hdptest-1 keytabs]# hdfs fetchdt --renewer hdfs my.delegation.token

16/08/04 22:21:05 WARN ipc.Client: Exception encountered while connecting to the server :

javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)

at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)

at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:563)

<output truncated for brevity>

Here you can see that delegation token can only be obtained based on the initial authentication with Kerberos. Failing to have the ticket would make Namenode to deny any Delegation token.

[#2]

Crisp and Clear...

[#3]

How to use this token with https://hdfscli.readthedocs.io/en/latest/api.html#hdfs.client.TokenClient