New Contributor
Posts: 2
Registered: ‎12-27-2013

MorphlineSolrSink: Unable to start SinkRunner java.lang.OutOfMemoryError: Java heap space



I'm trying to run the syslog example from Example Morphline Usage


However I get the following error right after starting flume with the command:


flume-ng agent --conf conf --conf-file flume/example.conf --name a1 -Dflume.root.logger=INFO,console

ERROR lifecycle.LifecycleSupervisor: Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@19484a05 counterGroup:{ name:null counters:{} } } - Exception follows.
java.lang.OutOfMemoryError: Java heap space
	at java.util.jar.JarFile$1.nextElement(
	at java.util.jar.JarFile$1.nextElement(
	at com.cloudera.cdk.morphline.api.MorphlineContext.getTopLevelClasses(
	at com.cloudera.cdk.morphline.api.MorphlineContext.importCommandBuilders(
	at com.cloudera.cdk.morphline.stdlib.Pipe.<init>(
	at com.cloudera.cdk.morphline.base.Compiler.compile(
	at com.cloudera.cdk.morphline.base.Compiler.compile(
	at org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl.configure(
	at org.apache.flume.sink.solr.morphline.MorphlineSink.start(
	at org.apache.flume.sink.DefaultSinkProcessor.start(
	at org.apache.flume.SinkRunner.start(
	at org.apache.flume.lifecycle.LifecycleSupervisor$
	at java.util.concurrent.Executors$
	at java.util.concurrent.FutureTask$Sync.innerRunAndReset(
	at java.util.concurrent.FutureTask.runAndReset(
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(
	at java.util.concurrent.ScheduledThreadPoolExecutor$
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
	at java.util.concurrent.ThreadPoolExecutor$
13/12/27 16:34:24 INFO morphline.MorphlineSink: Morphline Sink k2 stopping...

 I'm running CDH 4 on virtualBox, do I need to tune my flume config? am I short in memory? (4GB was allocated to vm) there's 1GB free memory before running flume.


Here's my flume conf file for refrence:

# Name the components on this agent
a1.sources = r1
a1.sinks = k1 k2
a1.channels = c1

# Describe/configure the source
#a1.sources.r1.type = netcat
#a1.sources.r1.bind = localhost
#a1.sources.r1.port = 44444

a1.sources.r1.type = syslogudp
a1.sources.r1.port = 5140 =

# Describe the sink
a1.sinks.k1.type = logger
a1.sinks.k2.type = org.apache.flume.sink.solr.morphline.MorphlineSolrSink
a1.sinks.k2.morphlineFile = /home/cloudera/flume/morphline-syslog.conf
a1.sinks.k2.batchSize = 100
a1.sinks.k2.batchDurationMillis = 1000
a1.sinks.k2.morphlineId = morphline1

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 1000

# Bind the source and sink to the channel
a1.sources.r1.channels = c1 = c1 = c1


Cloudera Employee
Posts: 146
Registered: ‎08-21-2013

Re: MorphlineSolrSink: Unable to start SinkRunner java.lang.OutOfMemoryError: Java heap space

Try to give the flume process more memory. I think it only has -Xmx 20 MB by default.