DataXceiver error processing REQUEST_SHORT_CIRCUIT_FDS operation

[#1]

I am getting the below error continuously in my logs, can you help me in solving the following error.

spectra.xxxx.sys.xxxxx.net:1004:DataXceiver error processing REQUEST_SHORT_CIRCUIT_FDS operation  src: unix:/var/run/hdfs-sockets/dn dst: <local>
org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with block_token_identifier (expiryDate=1566429148134, keyId=-1119489055, userId=hbase, blockPoolId=BP-1603344558-10.146.65.4-1468089654785, blockId=1815159924, access modes=[READ]) is expired.
	at org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
	at org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:301)
	at org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:97)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.checkAccess(DataXceiver.java:1289)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.requestShortCircuitFds(DataXceiver.java:295)
	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opRequestShortCircuitFds(Receiver.java:219)
	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:121)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:246)
	at java.lang.Thread.run(Thread.java:745)

[#2]

Hello @kal 

It looks like some of the DataNodes are not in sync, first try restarting your DataNodes. 

If you continue to receive the errors, please check the following:

1. Please check /etc/hosts file, it should be in sync on all the DataNodes and NameNodes if you are not using DNS.

2. Please check if iptables is running on few DataNodes, apply a for loop and quickly check on all the DataNodes.

3. Please check if time is in sync on all the DataNodes. Time on NameNode and Datanodes should be in sync.