Reply
Highlighted
Explorer
Posts: 69
Registered: ‎08-07-2017

Getting error while running flume agent

[ Edited ]

Hi,

 

I am getting below error while running the flume agent.

 

ERROR PollableSourceRunner:156 - Unhandled exception, logging and sleeping for 5000ms
org.apache.flume.ChannelException: Unable to put batch on required channel: FileChannel ch2 { dataDirs: [/var/lib/flume-ng/plugins.d/custom/datac2] }
at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
at org.keedio.flume.source.SQLSource$ChannelWriter.flush(SQLSource.java:195)
at org.keedio.flume.source.SQLSource$ChannelWriter.write(SQLSource.java:190)
at org.keedio.flume.source.SQLSource$ChannelWriter.write(SQLSource.java:190)
at java.io.Writer.write(Writer.java:192)
at java.io.PrintWriter.write(PrintWriter.java:456)
at java.io.PrintWriter.write(PrintWriter.java:473)
at com.opencsv.CSVWriter.writeNext(CSVWriter.java:263)
at com.opencsv.CSVWriter.writeAll(CSVWriter.java:151)
at org.keedio.flume.source.HibernateHelper.executeQuery(HibernateHelper.java:155)
at org.keedio.flume.source.SQLSource.process(SQLSource.java:127)
at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:139)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.ChannelFullException: The channel has reached it's capacity. This might be the result of a sink on the channel having too low of batch size,
a downstream system running slower than normal, or that the channel capacity is just too low. [channel=ch2]
at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)

 

The configuration settings that I am using for flume is as below.

 

agent1.channels.ch1.capacity=100000
agent1.channels.ch2.capacity=100000

agent1.sinks.sink1.morphlineId=morphline1
agent1.sinks.sink1.type=org.apache.flume.sink.solr.morphline.MorphlineSolrSink
agent1.sinks.sink1.channel=ch1
agent1.sinks.sink1.morphlineFile=/var/lib/flume-ng/plugins.d/custom/conf/morphlines.conf
agent1.sinks.sink1.batchSize=100
agent1.sinks.sink1.batchDurationMillis=1000

agent1.sinks.sink2.morphlineId=morphline1
agent1.sinks.sink2.type=org.apache.flume.sink.solr.morphline.MorphlineSolrSink
agent1.sinks.sink2.channel=ch2
agent1.sinks.sink2.morphlineFile=/var/lib/flume-ng/plugins.d/custom/conf/morphlines.conf
agent1.sinks.sink2.batchSize=100
agent1.sinks.sink2.batchDurationMillis=1000

agent1.sources=sql-source ods-sql-source
agent1.sinks=sink1 sink2
agent1.channels=ch1 ch2
#agent1.channels.ch1.type=memory
#agent1.channels.ch2.type=memory

agent1.sources.sql-source.type=org.keedio.flume.source.SQLSource
agent1.sources.ods-sql-source.type=org.keedio.flume.source.SQLSource

agent1.sources.sql-source.channels=ch1
agent1.sources.ods-sql-source.channels=ch2

#use FILE channel
agent1.channels.ch1.type = file
agent1.channels.ch1.transactionCapacity = 1000
agent1.channels.ch1.checkpointDir = /var/lib/flume-ng/plugins.d/custom/checkpointc1
#NOTE: point to your checkpoint directory
agent1.channels.ch1.dataDirs = /var/lib/flume-ng/plugins.d/custom/datac1
#NOTE: point to your data directory

#use FILE channel
agent1.channels.ch2.type = file
agent1.channels.ch2.transactionCapacity = 1000
agent1.channels.ch2.checkpointDir = /var/lib/flume-ng/plugins.d/custom/checkpointc2
#NOTE: point to your checkpoint directory
agent1.channels.ch2.dataDirs = /var/lib/flume-ng/plugins.d/custom/datac2
#NOTE: point to your data directory

 

 

 

Can you please help me on this?

 

Thanks,

Priya

Announcements