Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Warnings regarding Recovery Error

Highlighted

Warnings regarding Recovery Error

New Contributor

I recently replaced a failed disk on my datanode, and followed the steps to replace it. Stopped my cluster, edited the config file to remove the failed disk's directory, added a new disk, copied the files on the /dfs/dn to the new disk. restarted the cluster. The cluster health is good but when I used sqoop to ingest some tables, several warnings started popping out. The table was still ingested but the warnings have me worried cause I did a check and there were no corrupted or under-replicated blocks, so I am unsure on what is causing this.

 

I've attached the error message below.

Spoiler

19/09/05 16:00:58 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751368_10575

java.io.IOException: Bad response ERROR for BP-24464815-10.199.10.51-1567409513717:blk_1073751368_10575 from datanode DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1126)

19/09/05 16:00:58 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751368_10575 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK], DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]]: datanode 1(DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]) is bad.

19/09/05 16:00:58 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751370_10578

java.io.EOFException: Unexpected EOF while trying to read response from server

at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:539)

at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086)

19/09/05 16:00:58 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751370_10578 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]]: datanode 0(DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK]) is bad.

19/09/05 16:00:58 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751375_10584

java.io.IOException: Bad response ERROR for BP-24464815-10.199.10.51-1567409513717:blk_1073751375_10584 from datanode DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1126)

19/09/05 16:00:58 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751375_10584 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]]: datanode 1(DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]) is bad.

19/09/05 16:00:58 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751376_10586

java.io.IOException: Bad response ERROR for BP-24464815-10.199.10.51-1567409513717:blk_1073751376_10586 from datanode DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1126)

19/09/05 16:00:58 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751376_10586 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]]: datanode 2(DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]) is bad.

19/09/05 16:00:58 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751377_10588

java.io.EOFException: Unexpected EOF while trying to read response from server

at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:539)

at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086)

19/09/05 16:00:58 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751377_10588 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]]: datanode 0(DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK]) is bad.

19/09/05 16:00:58 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751378_10590

java.io.IOException: Bad response ERROR for BP-24464815-10.199.10.51-1567409513717:blk_1073751378_10590 from datanode DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1126)

19/09/05 16:00:58 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751378_10590 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]]: datanode 2(DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]) is bad.

19/09/05 16:00:59 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751384_10597

java.io.EOFException: Unexpected EOF while trying to read response from server

at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:539)

at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086)

19/09/05 16:00:59 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751384_10597 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK], DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]]: datanode 0(DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK]) is bad.

19/09/05 16:00:59 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751385_10599

java.io.IOException: Bad response ERROR for BP-24464815-10.199.10.51-1567409513717:blk_1073751385_10599 from datanode DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1126)

19/09/05 16:00:59 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751385_10599 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]]: datanode 1(DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]) is bad.

19/09/05 16:00:59 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751387_10602

java.io.IOException: Bad response ERROR for BP-24464815-10.199.10.51-1567409513717:blk_1073751387_10602 from datanode DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1126)

19/09/05 16:00:59 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751387_10602 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK], DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]]: datanode 1(DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]) is bad.

19/09/05 16:00:59 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751388_10604

java.io.IOException: Bad response ERROR for BP-24464815-10.199.10.51-1567409513717:blk_1073751388_10604 from datanode DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1126)

19/09/05 16:00:59 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751388_10604 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]]: datanode 1(DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]) is bad.

19/09/05 16:00:59 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751391_10608

java.io.EOFException: Unexpected EOF while trying to read response from server

at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:539)

at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086)

19/09/05 16:00:59 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751391_10608 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]]: datanode 0(DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK]) is bad.

19/09/05 16:01:00 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751396_10614

java.io.IOException: Bad response ERROR for BP-24464815-10.199.10.51-1567409513717:blk_1073751396_10614 from datanode DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1126)

19/09/05 16:01:00 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751396_10614 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]]: datanode 2(DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]) is bad.

19/09/05 16:01:00 INFO hdfs.DataStreamer: Exception in createBlockOutputStream blk_1073751396_10614

java.io.IOException: Got error, status=ERROR, status message , ack with firstBadLink as 10.199.10.51:9866

at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:134)

at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:110)

at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1778)

at org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1507)

at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1481)

at org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1256)

at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:667)

19/09/05 16:01:00 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751396_10614 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]]: datanode 1(DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]) is bad.

19/09/05 16:01:00 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751398_10618

java.io.EOFException: Unexpected EOF while trying to read response from server

at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:539)

at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086)

19/09/05 16:01:00 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751398_10618 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]]: datanode 0(DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK]) is bad.

19/09/05 16:01:00 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751400_10621

java.io.IOException: Bad response ERROR for BP-24464815-10.199.10.51-1567409513717:blk_1073751400_10621 from datanode DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1126)

19/09/05 16:01:00 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751400_10621 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]]: datanode 1(DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]) is bad.

19/09/05 16:01:00 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751405_10627

java.io.EOFException: Unexpected EOF while trying to read response from server

at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:539)

at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086)

19/09/05 16:01:00 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751405_10627 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]]: datanode 0(DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK]) is bad.

19/09/05 16:01:00 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751406_10629

java.io.IOException: Bad response ERROR for BP-24464815-10.199.10.51-1567409513717:blk_1073751406_10629 from datanode DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1126)

19/09/05 16:01:00 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751406_10629 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]]: datanode 2(DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]) is bad.

19/09/05 16:01:00 INFO hdfs.DataStreamer: Exception in createBlockOutputStream blk_1073751406_10629

java.io.IOException: Got error, status=ERROR, status message , ack with firstBadLink as 10.199.10.53:9866

at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:134)

at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:110)

at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1778)

at org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1507)

at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1481)

at org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1256)

at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:667)

19/09/05 16:01:00 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751406_10629 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]]: datanode 1(DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]) is bad.

19/09/05 16:01:00 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751411_10636

java.io.IOException: Bad response ERROR for BP-24464815-10.199.10.51-1567409513717:blk_1073751411_10636 from datanode DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1126)

19/09/05 16:01:00 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751411_10636 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]]: datanode 1(DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]) is bad.

19/09/05 16:01:00 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751412_10638

java.io.EOFException: Unexpected EOF while trying to read response from server

at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:539)

at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086)

19/09/05 16:01:00 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751412_10638 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]]: datanode 0(DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK]) is bad.

19/09/05 16:01:01 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751415_10642

java.io.IOException: Bad response ERROR for BP-24464815-10.199.10.51-1567409513717:blk_1073751415_10642 from datanode DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1126)

19/09/05 16:01:01 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751415_10642 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK], DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]]: datanode 1(DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]) is bad.

19/09/05 16:01:01 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751419_10647

java.io.IOException: Bad response ERROR for BP-24464815-10.199.10.51-1567409513717:blk_1073751419_10647 from datanode DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1126)

19/09/05 16:01:01 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751419_10647 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]]: datanode 2(DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]) is bad.

19/09/05 16:01:01 INFO hdfs.DataStreamer: Exception in createBlockOutputStream blk_1073751419_10647

java.io.EOFException: Unexpected EOF while trying to read response from server

at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:539)

at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1762)

at org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1507)

at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1481)

at org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1256)

at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:667)

19/09/05 16:01:01 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751419_10647 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]]: datanode 0(DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK]) is bad.

19/09/05 16:01:01 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751420_10650

java.io.IOException: Bad response ERROR for BP-24464815-10.199.10.51-1567409513717:blk_1073751420_10650 from datanode DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1126)

19/09/05 16:01:01 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751420_10650 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK], DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]]: datanode 2(DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]) is bad.

19/09/05 16:01:01 INFO db.DBInputFormat: Using read commited transaction isolation

19/09/05 16:01:01 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751426_10657

java.io.EOFException: Unexpected EOF while trying to read response from server

at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:539)

at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086)

19/09/05 16:01:01 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751426_10657 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]]: datanode 0(DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK]) is bad.