Getting "Twitter4J Async Dispatcher[0]" java.lang.OutOfMemoryError: Java heap space while streaming data from Twitter

[#1]

Getting "Twitter4J Async Dispatcher[0]" java.lang.OutOfMemoryError: Java heap space while streaming data from Twitter

ERROR channel.ChannelProcessor: Error while writing to required channel: FileChannel FileChannel { dataDirs: [/var/log/flume/data] }
java.lang.OutOfMemoryError: Java heap space
   at com.google.protobuf.ByteString.copyFrom(ByteString.java:192)
   at com.google.protobuf.ByteString.copyFrom(ByteString.java:204)
   at org.apache.flume.channel.file.Put.writeProtos(Put.java:89)
   at org.apache.flume.channel.file.TransactionEventRecord.toByteBuffer(TransactionEventRecord.java:174)
   at org.apache.flume.channel.file.Log.put(Log.java:642)
   at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:468)
   at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
   at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
   at org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:265)
   at org.apache.flume.source.twitter.TwitterSource.onStatus(TwitterSource.java:173)
   at twitter4j.StatusStreamImpl.onStatus(StatusStreamImpl.java:75)
   at twitter4j.StatusStreamBase$1.run(StatusStreamBase.java:114)
   at twitter4j.internal.async.ExecuteThread.run(DispatcherImpl.java:116)
Exception in thread "Twitter4J Async Dispatcher[0]" java.lang.OutOfMemoryError: Java heap space
   at com.google.protobuf.ByteString.copyFrom(ByteString.java:192)
   at com.google.protobuf.ByteString.copyFrom(ByteString.java:204)
   at org.apache.flume.channel.file.Put.writeProtos(Put.java:89)
   at org.apache.flume.channel.file.TransactionEventRecord.toByteBuffer(TransactionEventRecord.java:174)
   at org.apache.flume.channel.file.Log.put(Log.java:642)
   at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:468)
   at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
   at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
   at org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:265)
   at org.apache.flume.source.twitter.TwitterSource.onStatus(TwitterSource.java:173)
   at twitter4j.StatusStreamImpl.onStatus(StatusStreamImpl.java:75)
   at twitter4j.StatusStreamBase$1.run(StatusStreamBase.java:114)
   at twitter4j.internal.async.ExecuteThread.run(DispatcherImpl.java:116)

[#2]

can you show a Java dump? What are you memory settings? What is the JVM heap dump? What are your Flume settings and versions?

Is there space on /var/log/flume/data

See here:

https://issues.apache.org/jira/browse/FLUME-2403

Do the twitter messages before that look okay? Normal JSON?

Have you tried this in NIFI? My Twitter to NIFI worked without issue.

[#3]

@Timothy Spann

Flume version : 1.6.0 installed in HDP 2.4 using Ambari 2.2.1

Twitter messages are looking normal JSON.

I changed JAVA_OPTS value also but no luck.

mapreduce.map.java.opts is -Xmx1228m if i increase it is coming back to default automatically.

Please suggest. Thanks in advance.!

[#4 · Accepted solution]

The issue is with the role size and it has been resolved afterchanging the rollsize in the twitter conf file as below.

TwitterAgent.sinks.HDFS.hdfs.rollsize=67100884