Datanode Failures: DataXceiver error processing WRITE_BLOCK/READ_BLOCK operation

[#1]

I have been experiencing failures with my datanodes and the error is WRITE_BLOCK and READ_BLOCK. I have checked the data handlers and i have dfs.datanode.max.transfer.threads set to 16384. I run HDP 2.4.3 with 11 nodes. Please see error below;

2017-03-24 10:09:59,749 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dn:50010:DataXceiver error processing READ_BLOCK operation  src: /ip_address:49591 dst: /ip_address:50010

2017-03-24 11:02:18,750 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dn:50010:DataXceiver error processing WRITE_BLOCK operation  src: /ip_address:43052 dst: /ip_address:50010

[#2]

I'd also post this question on the Ambari track to check why Ambari didn't detect the DataNodes doing down.

Also from your logs it is hard to say why the DataNode went down. I again recommend increasing the DataNode heap allocation via Ambari. Also check that your nodes are provisioned with sufficient amount of RAM.