Reply
New Contributor
Posts: 2
Registered: ‎12-27-2013

MorphlineSolrSink: Unable to start SinkRunner java.lang.OutOfMemoryError: Java heap space

Hi,

 

I'm trying to run the syslog example from Example Morphline Usage

 

However I get the following error right after starting flume with the command:

 

flume-ng agent --conf conf --conf-file flume/example.conf --name a1 -Dflume.root.logger=INFO,console

ERROR lifecycle.LifecycleSupervisor: Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@19484a05 counterGroup:{ name:null counters:{} } } - Exception follows.
java.lang.OutOfMemoryError: Java heap space
	at java.util.jar.JarFile$1.nextElement(JarFile.java:226)
	at java.util.jar.JarFile$1.nextElement(JarFile.java:220)
	at com.cloudera.cdk.morphline.shaded.com.google.common.reflect.ClassPath$Scanner.scanJar(ClassPath.java:343)
	at com.cloudera.cdk.morphline.shaded.com.google.common.reflect.ClassPath$Scanner.scanFrom(ClassPath.java:288)
	at com.cloudera.cdk.morphline.shaded.com.google.common.reflect.ClassPath$Scanner.scan(ClassPath.java:276)
	at com.cloudera.cdk.morphline.shaded.com.google.common.reflect.ClassPath.from(ClassPath.java:84)
	at com.cloudera.cdk.morphline.api.MorphlineContext.getTopLevelClasses(MorphlineContext.java:100)
	at com.cloudera.cdk.morphline.api.MorphlineContext.importCommandBuilders(MorphlineContext.java:68)
	at com.cloudera.cdk.morphline.stdlib.Pipe.<init>(Pipe.java:41)
	at com.cloudera.cdk.morphline.stdlib.PipeBuilder.build(PipeBuilder.java:39)
	at com.cloudera.cdk.morphline.base.Compiler.compile(Compiler.java:125)
	at com.cloudera.cdk.morphline.base.Compiler.compile(Compiler.java:54)
	at org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl.configure(MorphlineHandlerImpl.java:90)
	at org.apache.flume.sink.solr.morphline.MorphlineSink.start(MorphlineSink.java:97)
	at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
	at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
	at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)
13/12/27 16:34:24 INFO morphline.MorphlineSink: Morphline Sink k2 stopping...

 I'm running CDH 4 on virtualBox, do I need to tune my flume config? am I short in memory? (4GB was allocated to vm) there's 1GB free memory before running flume.

 

Here's my flume conf file for refrence:

# Name the components on this agent
a1.sources = r1
a1.sinks = k1 k2
a1.channels = c1

# Describe/configure the source
#a1.sources.r1.type = netcat
#a1.sources.r1.bind = localhost
#a1.sources.r1.port = 44444

a1.sources.r1.type = syslogudp
a1.sources.r1.port = 5140
a1.sources.r1.host = 0.0.0.0


# Describe the sink
a1.sinks.k1.type = logger
a1.sinks.k2.type = org.apache.flume.sink.solr.morphline.MorphlineSolrSink
a1.sinks.k2.morphlineFile = /home/cloudera/flume/morphline-syslog.conf
a1.sinks.k2.batchSize = 100
a1.sinks.k2.batchDurationMillis = 1000
a1.sinks.k2.morphlineId = morphline1

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 1000

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c1

 

Highlighted
Cloudera Employee
Posts: 146
Registered: ‎08-21-2013

Re: MorphlineSolrSink: Unable to start SinkRunner java.lang.OutOfMemoryError: Java heap space

Try to give the flume process more memory. I think it only has -Xmx 20 MB by default.

Announcements