Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Getting "Twitter4J Async Dispatcher[0]" java.lang.OutOfMemoryError: Java heap space while streaming data from Twitter

Solved Go to solution
Highlighted

Getting "Twitter4J Async Dispatcher[0]" java.lang.OutOfMemoryError: Java heap space while streaming data from Twitter

Rising Star

Getting "Twitter4J Async Dispatcher[0]" java.lang.OutOfMemoryError: Java heap space while streaming data from Twitter

ERROR channel.ChannelProcessor: Error while writing to required channel: FileChannel FileChannel { dataDirs: [/var/log/flume/data] }
java.lang.OutOfMemoryError: Java heap space
   at com.google.protobuf.ByteString.copyFrom(ByteString.java:192)
   at com.google.protobuf.ByteString.copyFrom(ByteString.java:204)
   at org.apache.flume.channel.file.Put.writeProtos(Put.java:89)
   at org.apache.flume.channel.file.TransactionEventRecord.toByteBuffer(TransactionEventRecord.java:174)
   at org.apache.flume.channel.file.Log.put(Log.java:642)
   at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:468)
   at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
   at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
   at org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:265)
   at org.apache.flume.source.twitter.TwitterSource.onStatus(TwitterSource.java:173)
   at twitter4j.StatusStreamImpl.onStatus(StatusStreamImpl.java:75)
   at twitter4j.StatusStreamBase$1.run(StatusStreamBase.java:114)
   at twitter4j.internal.async.ExecuteThread.run(DispatcherImpl.java:116)
Exception in thread "Twitter4J Async Dispatcher[0]" java.lang.OutOfMemoryError: Java heap space
   at com.google.protobuf.ByteString.copyFrom(ByteString.java:192)
   at com.google.protobuf.ByteString.copyFrom(ByteString.java:204)
   at org.apache.flume.channel.file.Put.writeProtos(Put.java:89)
   at org.apache.flume.channel.file.TransactionEventRecord.toByteBuffer(TransactionEventRecord.java:174)
   at org.apache.flume.channel.file.Log.put(Log.java:642)
   at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:468)
   at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
   at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
   at org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:265)
   at org.apache.flume.source.twitter.TwitterSource.onStatus(TwitterSource.java:173)
   at twitter4j.StatusStreamImpl.onStatus(StatusStreamImpl.java:75)
   at twitter4j.StatusStreamBase$1.run(StatusStreamBase.java:114)
   at twitter4j.internal.async.ExecuteThread.run(DispatcherImpl.java:116)
1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Getting "Twitter4J Async Dispatcher[0]" java.lang.OutOfMemoryError: Java heap space while streaming data from Twitter

Rising Star

The issue is with the role size and it has been resolved afterchanging the rollsize in the twitter conf file as below.

TwitterAgent.sinks.HDFS.hdfs.rollsize=67100884

View solution in original post

3 REPLIES 3
Highlighted

Re: Getting "Twitter4J Async Dispatcher[0]" java.lang.OutOfMemoryError: Java heap space while streaming data from Twitter

Super Guru

can you show a Java dump? What are you memory settings? What is the JVM heap dump? What are your Flume settings and versions?

Is there space on /var/log/flume/data

See here:

https://issues.apache.org/jira/browse/FLUME-2403

Do the twitter messages before that look okay? Normal JSON?

Have you tried this in NIFI? My Twitter to NIFI worked without issue.

Highlighted

Re: Getting "Twitter4J Async Dispatcher[0]" java.lang.OutOfMemoryError: Java heap space while streaming data from Twitter

Rising Star

@Timothy Spann

Flume version : 1.6.0 installed in HDP 2.4 using Ambari 2.2.1

Twitter messages are looking normal JSON.

I changed JAVA_OPTS value also but no luck.

mapreduce.map.java.opts is -Xmx1228m if i increase it is coming back to default automatically.

Please suggest. Thanks in advance.!

Highlighted

Re: Getting "Twitter4J Async Dispatcher[0]" java.lang.OutOfMemoryError: Java heap space while streaming data from Twitter

Rising Star

The issue is with the role size and it has been resolved afterchanging the rollsize in the twitter conf file as below.

TwitterAgent.sinks.HDFS.hdfs.rollsize=67100884

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here