Reply
Explorer
Posts: 13
Registered: ‎04-20-2017

Flume FileChannel FileNotFoundException

[ Edited ]

Hi,

 

I changed a Flume agent to use FileChannel instead of MemoryChannel. The agent reads from Kafka and writes to HDFS. It ran fine for a few days, then suddenly I got this error message:

2017-11-30 08:53:45,741 (Log-BackgroundWorker-my_file_channel) [ERROR - org.apache.flume.channel.file.Log$BackgroundWorker.run(Log.java:1238)] Error doing checkpoint
java.io.FileNotFoundException: /var/lib/hadoop-hdfs/.flume/file-channel/data/log-30 (No such file or directory)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)
        at org.apache.flume.channel.file.LogFileFactory.getMetaDataWriter(LogFileFactory.java:48)
        at org.apache.flume.channel.file.Log.writeCheckpoint(Log.java:1057)
        at org.apache.flume.channel.file.Log.writeCheckpoint(Log.java:1015)
        at org.apache.flume.channel.file.Log.access$200(Log.java:76)
        at org.apache.flume.channel.file.Log$BackgroundWorker.run(Log.java:1234)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

Since then, Flume logs this message every 5 seconds:

2017-12-04 07:39:05,215 (PollableSourceRunner-KafkaSource-my_kafka_source) [ERROR - org.apache.flume.source.kafka.KafkaSource.doProcess(KafkaSource.java:314)] KafkaSource EXCEPTION, {}
org.apache.flume.ChannelException: Commit failed due to IO error [channel=my_file_channel]
        at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doRollback(FileChannel.java:660)
        at org.apache.flume.channel.BasicTransactionSemantics.rollback(BasicTransactionSemantics.java:168)
        at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:196)
        at org.apache.flume.source.kafka.KafkaSource.doProcess(KafkaSource.java:295)
        at org.apache.flume.source.AbstractPollableSource.process(AbstractPollableSource.java:60)
        at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:133)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Usable space exhausted, only 0 bytes remaining, required 524288026 bytes
        at org.apache.flume.channel.file.Log.rollback(Log.java:744)
        at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doRollback(FileChannel.java:658)
        ... 6 more

I'm convinced that the "Usable space exhausted" can't be the real issue, since the disk is only 5% full.

batchSize and channel transactionCapacity are set to 1000, channel capacity is 10000.

The number of events are relatively low, max 50/sec.

Any idea what could be the issue here?

 

Thanks!

 

 

Announcements