Reply
dsp
New Contributor
Posts: 1
Registered: ‎04-11-2014

flume HDFS write errors

Hello

 

We experience some troubles with flume writing to hdfs.

 

in flume logs we can see lines containning :

 

 

11 Apr 2014 00:23:30,901 ERROR [hdfs-eventSink-call-runner-22] (org.apache.flume.sink.hdfs.AbstractHDFSWriter.closeHDFSOutputStream:267)  - Unable to close HDFS file: 'hdfs://_events-21.1397168460389.avro.tmp'

 

11 Apr 2014 00:23:33,295 WARN  [ResponseProcessor for block BP-61758603-10.194.97.1-1366380297815:blk_2482324759628484456_30836486] (org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run:747)  - DFSOutputStream ResponseProcessor exception  for block BP-61758603-10.194.97.1-1366380297815:blk_2482324759628484456_30836486

java.net.SocketTimeoutException: 75000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.194.8.67:19255 remote=/10.194.98.17:50010]

        at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:165)

        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:156)

        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:129)

        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:117)

        at java.io.FilterInputStream.read(FilterInputStream.java:83)

        at java.io.FilterInputStream.read(FilterInputStream.java:83)

        at org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:169)

        at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:114)

        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:694)

 

11 Apr 2014 00:23:33,296 WARN  [DataStreamer for file /_events-22.1397168537969.avro.tmp block BP-61758603-10.194.97.1-1366380297815:blk_2482324759628484456_30836486] (org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery:964)  - Error Recovery for block BP-61758603-10.194.97.1-1366380297815:blk_2482324759628484456_30836486 in pipeline 10.194.98.17:50010, 10.194.98.14:50010, 10.194.98.12:50010: bad datanode 10.194.98.17:50010

11 Apr 2014 00:23:36,371 WARN  [SinkRunner-PollingRunner-FailoverSinkProcessor] (org.apache.flume.sink.hdfs.HDFSEventSink.process:418)  - HDFS IO error

java.io.IOException: Callable timed out after 30000 ms on file: hdfs://_events-22.1397168520957.avro.tmp

        at org.apache.flume.sink.hdfs.BucketWriter.callWithTimeout(BucketWriter.java:571)

        at org.apache.flume.sink.hdfs.BucketWriter.doFlush(BucketWriter.java:353)

        at org.apache.flume.sink.hdfs.BucketWriter.flush(BucketWriter.java:319)

        at org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:463)

        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:392)

        at org.apache.flume.sink.FailoverSinkProcessor.process(FailoverSinkProcessor.java:182)

        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)

        at java.lang.Thread.run(Thread.java:744)

Caused by: java.util.concurrent.TimeoutException

        at java.util.concurrent.FutureTask.get(FutureTask.java:201)

        at org.apache.flume.sink.hdfs.BucketWriter.callWithTimeout(BucketWriter.java:564)

        ... 7 more

 

 

This errors seems causing flumes problems / freezes / slow downs  resolved by restarting flume process.

 

We use flume version 1.4.0+56-1.cdh4.5.0.p0.16~precise-cdh4.5.0

with hadoop cluster version 2.0.0+1357-1.cdh4.3.0.p0.21~precise-cdh4.3.0

 

 

 

Best regards

Highlighted
Contributor
Posts: 27
Registered: ‎08-26-2013

Re: flume HDFS write errors

We had something simliar.

 

We increased the size of the channel and increase the sink flush settings to write to HDFS fewer and larger files.

 

We were getting timeouts on the HDFS from too many small files.

Announcements
New solutions