question Re: [Flume] - GC overhead limit exceeded error in flume channel in Archives of Support Questions (Read Only)

[Flume] - GC overhead limit exceeded error in flume channel

osman — Fri, 16 Sep 2022 09:31:51 GMT

Hi, I’m working with CDH 5.4.2 and we use flume for data aggregation.When we ran the flume agent, observed the following problem:

18:06:2015 11:26:17 ERROR [ChannelProcessor] Error while writing to required channel: org.apache.flume.channel.MemoryChannel{name: channel1}
java.lang.OutOfMemoryError: GC overhead limit exceeded

Configuration File details:

agent1.channels.channel1.type = memory
agent1.channels.channel1.capacity = 100000
agent1.channels.channel1.transactionCapactiy = 1000

agent1.sources.source1.channels = channel1 

agent1.sinks = sink1

agent1.sinks.sink1.type = hdfs
agent1.sinks.sink1.channel = channel1
agent1.sinks.sink1.hdfs.useLocalTimeStamp = true
agent1.sinks.sink1.hdfs.path = hdfs://xxx/%Y%m%d%H
agent1.sinks.sink1.hdfs.filePrefix = xdr
agent1.sinks.sink1.hdfs.fileType = DataStream
agent1.sinks.sink1.hdfs.writeFormat = Text
agent1.sinks.sink1.hdfs.rollCount = 2000000
agent1.sinks.sink1.hdfs.rollSize = 0
agent1.sinks.sink1.hdfs.batchSize = 100

I really appreciate your help, on this issue.

Regards,

Osman Uygar

Re: [Flume] - GC overhead limit exceeded error in flume channel

Harsh J — Wed, 15 Jul 2015 09:28:49 GMT

You are using the memory channel, which sits inside the Java Heap of the Flume process. Your capacity indicates 100k items, so the overall usage if the channel gets full (based on the source's event sizes) can exhaust the heap if there's not sufficient overhead.

What is your current Flume heap size? You may want to double it, or estimate better the size based on average source event size times the channel capacity, plus some breathing room for the Flume daemon's own work.

Re: [Flume] - GC overhead limit exceeded error in flume channel

osman — Wed, 15 Jul 2015 10:51:16 GMT

We resolved this problem when adding below line to flume-env.sh;

export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote"

Thanks for your response

Re: [Flume] - GC overhead limit exceeded error in flume channel

Harsh J — Wed, 15 Jul 2015 10:55:37 GMT

Glad to know! Please consider marking your post as a solution so it helps others with similar issues arrive at a resolving post quicker.

Re: [Flume] - GC overhead limit exceeded error in flume channel

Manish99 — Mon, 22 May 2017 19:21:31 GMT

hi ,

i am also facing same issue .. but in my case it is not working as per your suggestion .. please help me still facing the same issue ..

ERROR hdfs.HDFSEventSink: process failed
java.lang.OutOfMemoryError: GC overhead limit exceeded

Re: [Flume] - GC overhead limit exceeded error in flume channel

vasuks — Mon, 18 Feb 2019 21:07:42 GMT

try giving the heap size when you start the flume agent

flume-ng agent -n a1 -f flume_config.conf -Xms256m -Xmx1024m