The Untold Story of Block Access Token

[#1]

Tokens are wire-serializable objects issued by Hadoop services, which grant access to services. Some services issue tokens to callers which are then used by those callers to directly interact with other services without involving the KDC at all.

Block Tokens

A BlockToken is the token issued for access to a block; it includes:

(userId, (BlockPoolId, BlockId), keyId, expiryDate, access-modes)

Block Keys

Key used for generating and verifying block tokens. Block Keys are managed in the BlockTokenSecretManager, one in the NN and another in every DN to track the block keys to which it has access.

How this Works:

1. Client asks NN for access to a path, identifying via Kerberos or delegation token.

2. Client talks to DNs with the block, using the Block Token.

3. DN authenticates Block Token using shared-secret with NameNode.

4. if authenticated, DN compares permissions in Block Token with operation requested, then grants or rejects the request.

The client does not have its identity checked by the DNs. That is done by the NN. This means that the client can in theory pass a Block Token on to another process for delegated access to a single block.

These HDFS Block Tokens do not contain any specific knowledge of the principal running the Datanodes, instead they declare that the caller has stated access rights to the specific block, up until the token expires.

public class BlockTokenIdentifier extends TokenIdentifier {
  static final Text KIND_NAME = new Text("HDFS_BLOCK_TOKEN");

  private long expiryDate;
  private int keyId;
  private String userId;
  private String blockPoolId;
  private long blockId;
  private final EnumSet<AccessMode> modes;
  private byte [] cache;

  ...

To enable the NameNode block access token, configure the following settings in the hdfs-site.xml file:

dfs.block.access.token.enable=yes
dfs.block.access.key.update.interval=600 (by default, minutes)
dfs.block.access.token.lifetime=600 (by default, minutes)

General Error Seen:

2015-09-22 12:55:48,271 WARN [regionserver60020-smallCompactions-1432895622947] shortcircuit.ShortCircuitCache: ShortCircuitCache(0x1102b41c): could not load 1074240550_BP-607492251-xxx.xxx.xxx.xxx-1427711497172 due to InvalidToken exception. 
org.apache.hadoop.security.token.SecretManager$InvalidToken: access control error while attempting to set up short-circuit access to /apps/hbase/data/data/default/blah/b83abaf5631c4ce18c9da7eaf569bb3b/t/bbb2436ed50e471e8645f8bd402902e3Block token with block_token_identifier (expiryDate=1442911790388, keyId=286785309, userId=hbase, blockPoolId=BP-607492251-xx.xx.xx.xx-1427711497172, blockId=1074240550, access modes=[READ]) is expired. 

Root Cause: The block token access is expire and become invalid

2018-07-15 17:49:25,649 WARN datanode.DataNode (DataXceiver.java:checkAccess(1311)) - 
Block token verification failed: op=WRITE_BLOCK, remoteAddress=/10.10.10.100:0000, 
message=Can't re-compute password for block_token_identifier (expiryDate=1501487365624, 
keyId=127533694, userId=RISHI, blockPoolId=BP-2019616911-10.211.159.22-1464205355083, 
blockId=1305095824, access modes=[WRITE]), since the required block key (keyID=127533694) 
doesn't exist.

Root Cause : This can be seen when a client connection fails because the client has presented a block access token that references a block key that does not exist in DataNode. To solve this restart the dataNode