Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

DataXceiver error processing REQUEST_SHORT_CIRCUIT_FDS operation

DataXceiver error processing REQUEST_SHORT_CIRCUIT_FDS operation

Explorer

I am getting the below error continuously in my logs, can you help me in solving the following error.

spectra.xxxx.sys.xxxxx.net:1004:DataXceiver error processing REQUEST_SHORT_CIRCUIT_FDS operation  src: unix:/var/run/hdfs-sockets/dn dst: <local>
org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with block_token_identifier (expiryDate=1566429148134, keyId=-1119489055, userId=hbase, blockPoolId=BP-1603344558-10.146.65.4-1468089654785, blockId=1815159924, access modes=[READ]) is expired.
	at org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
	at org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:301)
	at org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:97)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.checkAccess(DataXceiver.java:1289)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.requestShortCircuitFds(DataXceiver.java:295)
	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opRequestShortCircuitFds(Receiver.java:219)
	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:121)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:246)
	at java.lang.Thread.run(Thread.java:745)
1 REPLY 1
Highlighted

Re: DataXceiver error processing REQUEST_SHORT_CIRCUIT_FDS operation

Cloudera Employee

Hello @kal 

 

It looks like some of the DataNodes are not in sync, first try restarting your DataNodes. 

 

If you continue to receive the errors, please check the following:


1. Please check /etc/hosts file, it should be in sync on all the DataNodes and NameNodes if you are not using DNS.

 

2. Please check if iptables is running on few DataNodes, apply a for loop and quickly check on all the DataNodes.

 

3. Please check if time is in sync on all the DataNodes. Time on NameNode and Datanodes should be in sync.

Don't have an account?
Coming from Hortonworks? Activate your account here