Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Warnings regarding Recovery Error

Highlighted

Warnings regarding Recovery Error

I recently replaced a failed disk on my datanode, and followed the steps to replace it. Stopped my cluster, edited the config file to remove the failed disk's directory, added a new disk, copied the files on the /dfs/dn to the new disk. restarted the cluster. The cluster health is good but when I used sqoop to ingest some tables, several warnings started popping out. The table was still ingested but the warnings have me worried cause I did a check and there were no corrupted or under-replicated blocks, so I am unsure on what is causing this.

 

I've attached the error message below.

Spoiler

19/09/05 16:00:58 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751368_10575

java.io.IOException: Bad response ERROR for BP-24464815-10.199.10.51-1567409513717:blk_1073751368_10575 from datanode DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1126)

19/09/05 16:00:58 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751368_10575 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK], DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]]: datanode 1(DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]) is bad.

19/09/05 16:00:58 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751370_10578

java.io.EOFException: Unexpected EOF while trying to read response from server

at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:539)

at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086)

19/09/05 16:00:58 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751370_10578 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]]: datanode 0(DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK]) is bad.

19/09/05 16:00:58 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751375_10584

java.io.IOException: Bad response ERROR for BP-24464815-10.199.10.51-1567409513717:blk_1073751375_10584 from datanode DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1126)

19/09/05 16:00:58 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751375_10584 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]]: datanode 1(DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]) is bad.

19/09/05 16:00:58 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751376_10586

java.io.IOException: Bad response ERROR for BP-24464815-10.199.10.51-1567409513717:blk_1073751376_10586 from datanode DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1126)

19/09/05 16:00:58 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751376_10586 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]]: datanode 2(DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]) is bad.

19/09/05 16:00:58 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751377_10588

java.io.EOFException: Unexpected EOF while trying to read response from server

at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:539)

at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086)

19/09/05 16:00:58 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751377_10588 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]]: datanode 0(DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK]) is bad.

19/09/05 16:00:58 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751378_10590

java.io.IOException: Bad response ERROR for BP-24464815-10.199.10.51-1567409513717:blk_1073751378_10590 from datanode DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1126)

19/09/05 16:00:58 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751378_10590 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]]: datanode 2(DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]) is bad.

19/09/05 16:00:59 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751384_10597

java.io.EOFException: Unexpected EOF while trying to read response from server

at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:539)

at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086)

19/09/05 16:00:59 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751384_10597 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK], DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]]: datanode 0(DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK]) is bad.

19/09/05 16:00:59 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751385_10599

java.io.IOException: Bad response ERROR for BP-24464815-10.199.10.51-1567409513717:blk_1073751385_10599 from datanode DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1126)

19/09/05 16:00:59 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751385_10599 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]]: datanode 1(DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]) is bad.

19/09/05 16:00:59 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751387_10602

java.io.IOException: Bad response ERROR for BP-24464815-10.199.10.51-1567409513717:blk_1073751387_10602 from datanode DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1126)

19/09/05 16:00:59 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751387_10602 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK], DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]]: datanode 1(DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]) is bad.

19/09/05 16:00:59 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751388_10604

java.io.IOException: Bad response ERROR for BP-24464815-10.199.10.51-1567409513717:blk_1073751388_10604 from datanode DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1126)

19/09/05 16:00:59 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751388_10604 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]]: datanode 1(DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]) is bad.

19/09/05 16:00:59 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751391_10608

java.io.EOFException: Unexpected EOF while trying to read response from server

at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:539)

at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086)

19/09/05 16:00:59 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751391_10608 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]]: datanode 0(DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK]) is bad.

19/09/05 16:01:00 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751396_10614

java.io.IOException: Bad response ERROR for BP-24464815-10.199.10.51-1567409513717:blk_1073751396_10614 from datanode DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1126)

19/09/05 16:01:00 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751396_10614 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]]: datanode 2(DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]) is bad.

19/09/05 16:01:00 INFO hdfs.DataStreamer: Exception in createBlockOutputStream blk_1073751396_10614

java.io.IOException: Got error, status=ERROR, status message , ack with firstBadLink as 10.199.10.51:9866

at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:134)

at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:110)

at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1778)

at org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1507)

at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1481)

at org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1256)

at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:667)

19/09/05 16:01:00 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751396_10614 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]]: datanode 1(DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]) is bad.

19/09/05 16:01:00 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751398_10618

java.io.EOFException: Unexpected EOF while trying to read response from server

at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:539)

at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086)

19/09/05 16:01:00 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751398_10618 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]]: datanode 0(DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK]) is bad.

19/09/05 16:01:00 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751400_10621

java.io.IOException: Bad response ERROR for BP-24464815-10.199.10.51-1567409513717:blk_1073751400_10621 from datanode DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1126)

19/09/05 16:01:00 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751400_10621 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]]: datanode 1(DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]) is bad.

19/09/05 16:01:00 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751405_10627

java.io.EOFException: Unexpected EOF while trying to read response from server

at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:539)

at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086)

19/09/05 16:01:00 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751405_10627 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]]: datanode 0(DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK]) is bad.

19/09/05 16:01:00 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751406_10629

java.io.IOException: Bad response ERROR for BP-24464815-10.199.10.51-1567409513717:blk_1073751406_10629 from datanode DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1126)

19/09/05 16:01:00 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751406_10629 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]]: datanode 2(DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]) is bad.

19/09/05 16:01:00 INFO hdfs.DataStreamer: Exception in createBlockOutputStream blk_1073751406_10629

java.io.IOException: Got error, status=ERROR, status message , ack with firstBadLink as 10.199.10.53:9866

at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:134)

at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:110)

at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1778)

at org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1507)

at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1481)

at org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1256)

at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:667)

19/09/05 16:01:00 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751406_10629 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]]: datanode 1(DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]) is bad.

19/09/05 16:01:00 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751411_10636

java.io.IOException: Bad response ERROR for BP-24464815-10.199.10.51-1567409513717:blk_1073751411_10636 from datanode DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1126)

19/09/05 16:01:00 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751411_10636 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]]: datanode 1(DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]) is bad.

19/09/05 16:01:00 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751412_10638

java.io.EOFException: Unexpected EOF while trying to read response from server

at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:539)

at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086)

19/09/05 16:01:00 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751412_10638 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]]: datanode 0(DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK]) is bad.

19/09/05 16:01:01 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751415_10642

java.io.IOException: Bad response ERROR for BP-24464815-10.199.10.51-1567409513717:blk_1073751415_10642 from datanode DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1126)

19/09/05 16:01:01 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751415_10642 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK], DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]]: datanode 1(DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK]) is bad.

19/09/05 16:01:01 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751419_10647

java.io.IOException: Bad response ERROR for BP-24464815-10.199.10.51-1567409513717:blk_1073751419_10647 from datanode DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1126)

19/09/05 16:01:01 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751419_10647 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]]: datanode 2(DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]) is bad.

19/09/05 16:01:01 INFO hdfs.DataStreamer: Exception in createBlockOutputStream blk_1073751419_10647

java.io.EOFException: Unexpected EOF while trying to read response from server

at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:539)

at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1762)

at org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1507)

at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1481)

at org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1256)

at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:667)

19/09/05 16:01:01 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751419_10647 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]]: datanode 0(DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK]) is bad.

19/09/05 16:01:01 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751420_10650

java.io.IOException: Bad response ERROR for BP-24464815-10.199.10.51-1567409513717:blk_1073751420_10650 from datanode DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1126)

19/09/05 16:01:01 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751420_10650 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK], DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]]: datanode 2(DatanodeInfoWithStorage[10.199.10.52:9866,DS-0a908c24-8202-4cdf-ae5a-c8588c313f67,DISK]) is bad.

19/09/05 16:01:01 INFO db.DBInputFormat: Using read commited transaction isolation

19/09/05 16:01:01 WARN hdfs.DataStreamer: Exception for BP-24464815-10.199.10.51-1567409513717:blk_1073751426_10657

java.io.EOFException: Unexpected EOF while trying to read response from server

at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:539)

at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)

at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086)

19/09/05 16:01:01 WARN hdfs.DataStreamer: Error Recovery for BP-24464815-10.199.10.51-1567409513717:blk_1073751426_10657 in pipeline [DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK], DatanodeInfoWithStorage[10.199.10.53:9866,DS-987394b9-4c34-4547-832d-31bdb43ae2a3,DISK], DatanodeInfoWithStorage[10.199.10.51:9866,DS-d3a5fe92-f946-43a9-bf1d-cd379857f02e,DISK]]: datanode 0(DatanodeInfoWithStorage[10.199.10.54:9866,DS-8927eadb-b840-4f24-b8cb-3a675a34034d,DISK]) is bad.

Don't have an account?
Coming from Hortonworks? Activate your account here