spark job gives " java.io.EOFException: Unexpected EOF while trying to read response from server " error

[#1]

pache.hadoop.hdfs.DataStreamer run - Exception for BP-890279762-147.22.18.13-1450824261954:blk_1215757593_143649958
java.io.EOFException: Unexpected EOF while trying to read response from server
at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:539)

Hi ,

Spark job give the below error please provide an solution to fix this.

at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)
at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086)
[WARN] 2021-02-07 08:16:22,098 org.apache.hadoop.hdfs.DataStreamer handleBadDatanode - Error Recovery for BP-890279762-147.22.18.13-1450824261954:blk_1215757593_143649958 in pipeline [DatanodeInfoWithStorage[147.22.18.1:1004,DS-32c91853-cdae-4020-99fa-40a22a96830a,DISK], DatanodeInfoWithStorage[147.22.18.22:1004,DS-4f7f5c87-7790-4419-9b77-bbad65dd5a00,DISK], DatanodeInfoWithStorage[147.22.18.20:1004,DS-db06ea47-d7fa-488a-8b00-4ac6a0773df4,DISK]]: datanode 0(DatanodeInfoWithStorage[147.22.18.1:1004,DS-32c91853-cdae-4020-99fa-40a22a96830a,DISK]) is bad.

[#2]

@moni Is the data node healthy? This seems issue with DN is not healthy. 

[#3]

we did check with the Unix OS team  and net work team .. they say the data nodes are healthy

[#4]

@moni In that case we might have to take a look at full log file. There could be some other information.