Hi, Guys while running my flume agents getting some error Here My error: Avro source source1: Unable to process event batch. Exception follows. org.apache.flume.ChannelException: Unable to put batch on required channel: org.apache.flume.channel.MemoryChannel{name: channel1} at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200) at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:386) at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:91) at org.apache.avro.ipc.Responder.respond(Responder.java:151) at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:310) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: org.apache.flume.ChannelFullException: Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the buffer size is too tight at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:130) at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151) at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192) ... 30 more and here my config files my local config: agent.sources = localsource agent.channels = memoryChannel agent.sinks = avro_Sink agent.sources.localsource.type = spooldir #agent.sources.localsource.shell = /bin/bash -c agent.sources.localsource.spoolDir = /home/dwh/teja/Flumedata/ agent.sources.localsource.fileHeader = true # The channel can be defined as follows. agent.sources.localsource.channels = memoryChannel # Each sink's type must be defined agent.sinks.avro_Sink.type = avro agent.sinks.avro_Sink.hostname=192.168.4.110 agent.sinks.avro_Sink.port= 8021 agent.sinks.avro_Sink.avro.batchSize = 100 agent.sinks.avro_Sink.avro.rollCount = 0 agent.sinks.avro_Sink.avro.rollSize = 73060831 agent.sinks.avro_Sink.avro.rollInterval = 0 agent.sources.localsource.interceptors = search-replace agent.sources.localsource.interceptors.search-replace.type = search_replace # Remove leading alphanumeric characters in an event body. agent.sources.localsource.interceptors.search-replace.searchPattern = ###|## agent.sources.localsource.interceptors.search-replace.replaceString = | #Specify the channel the sink should use agent.sinks.avro_Sink.channel = memoryChannel # Each channel's type is defined. agent.channels.memoryChannel.type = memory agent.channels.memoryChannel.capacity = 10000 agent.channels.memoryChannel.transactionCapacity = 1000 my server X config file tier1.sources = source1 tier1.channels = channel1 tier1.sinks = sink1 tier1.sources.source1.type = avro tier1.sources.source1.bind= 192.168.4.110 tier1.sources.source1.port= 8021 tier1.sources.source1.channels = channel1 tier1.channels.channel1.type= memory tier1.sinks.sink1.type = hdfs tier1.sinks.sink1.channel = channel1 tier1.sinks.sink1.hdfs.path = hdfs://192.168.4.110:8021/user/hadoop/flumelogs/ tier1.sinks.sink1.hdfs.fileType = DataStream tier1.sinks.sink1.hdfs.writeFormat= Text tier1.sinks.sink1.hdfs.batchSize = 100 tier1.sinks.sink1.hdfs.rollCount = 0 tier1.sinks.sink1.hdfs.rollSize = 73060831 tier1.sinks.sink1.hdfs.rollInterval = 0 tier1.channels.channel1.capacity = 10000 tier1.channels.channel1.transactionCapacity = 1000   Please help if any one familiar with this.   Thanks in advance. ... View more

Hi, Guys              I want to drop full record by searching a key using interceptor in flume is that possible to drop full record in flume?   For example in my log i have record like : (this record in pagenotfound.php) by searching pagenotfound.php  this keyword i want to drop that full record is that possible ?   Please Help Thanks in advance ... View more

Im trying to movie log files from server X to hdfs. while moving i want to filter or replace some data, so for that i'm using search&replace interceptor. But it not working Please Help   Here My config file   Local Agent: agent.sources = localsource agent.channels = memoryChannel agent.sinks = avro_Sink   agent.sources.localsource.type = spooldir #agent.sources.localsource.shell = /bin/bash -c agent.sources.localsource.spoolDir = /home/dwh/teja/Flumedata/ agent.sources.localsource.fileHeader = true # The channel can be defined as follows. agent.sources.localsource.channels = memoryChannel # Each sink's type must be defined agent.sinks.avro_Sink.type = avro agent.sinks.avro_Sink.hostname=192.168.4.444 agent.sinks.avro_Sink.port= 8021 agent.sinks.avro_Sink.avro.batchSize = 100 agent.sinks.avro_Sink.avro.rollCount = 0 agent.sinks.avro_Sink.avro.rollSize = 65000000 agent.sinks.avro_Sink.avro.rollInterval = 0 agent.sources.localsource.interceptors = search-replace agent.sources.localsource.interceptors.search-replace.type = search_replace # Remove leading alphanumeric characters in an event body. agent.sources.localsource.interceptors.search-replace.searchPattern = '/^\s*\#+|\#+$/' agent.sources.localsource.interceptors.search-replace.replaceString = '|' #Specify the channel the sink should use agent.sinks.avro_Sink.channel = memoryChannel # Each channel's type is defined. agent.channels.memoryChannel.type = memory agent.channels.memoryChannel.capacity = 10000 agent.channels.memoryChannel.transactionCapacity = 1000   Here My Hdfs Agent: tier1.sources = source1 tier1.channels = channel1 tier1.sinks = sink1 # For each source, channel, and sink, set # standard properties. tier1.sources.source1.type = avro tier1.sources.source1.bind = 192.168.4.51 tier1.sources.source1.port=8021 tier1.sources.source1.channels = channel1 tier1.channels.channel1.type = memory tier1.sinks.sink1.type = hdfs tier1.sinks.sink1.channel = channel1 tier1.sinks.sink1.hdfs.path = hdfs://192.168.4.51:8020/user/hadoop/flumelogs/ tier1.sinks.sink1.hdfs.fileType = DataStream tier1.sinks.sink1.hdfs.writeFormat= Text tier1.sinks.sink1.hdfs.batchSize = 100 tier1.sinks.sink1.hdfs.rollCount = 0 tier1.sinks.sink1.hdfs.rollSize = 65000000 tier1.sinks.sink1.hdfs.rollInterval = 0 tier1.channels.channel1.capacity = 10000 tier1.channels.channel1.transactioncapacity=1000   Please say any thing need to add Thanks in advance   ... View more

I'm trying to load data from local to hdfs using spooldir source getting process failed error   Here My error process failed org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:96) at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113) at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95) at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:374) at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) at java.lang.Thread.run(Thread.java:745)   and here my confi file   Local agent   agent.sources.localsource.type = spooldir #agent.sources.localsource.shell = /bin/bash -c agent.sources.localsource.spoolDir = /home/dwh/teja/Flumedata/ agent.sources.localsource.fileHeader = true # The channel can be defined as follows. agent.sources.localsource.channels = memoryChannel # Each sink's type must be defined agent.sinks.avro_Sink.type = avro agent.sinks.avro_Sink.hostname=192.168.4.444 agent.sinks.avro_Sink.port= 8021 agent.sinks.avro_Sink.avro.batchSize = 1000 agent.sinks.avro_Sink.avro.rollCount = 0 agent.sinks.avro_Sink.avro.rollSize = 1000000 agent.sinks.avro_Sink.avro.rollInterval = 0 agent.sinks.avro_Sink.channel = memoryChannel # Each channel's type is defined. agent.channels.memoryChannel.type = memory # In this case, it specifies the capacity of the memory channel agent.channels.memoryChannel.capacity = 10000 agent.channels.memoryChannel.transactionCapacity = 10000   Remote config file   # Please paste flume.conf here. Example: # Sources, channels, and sinks are defined per # agent name, in this case 'tier1'. tier1.sources = source1 tier1.channels = channel1 tier1.sinks = sink1 tier1.sources.source1.type = avro tier1.sources.source1.bind = 192.168.4.444 tier1.sources.source1.port=8021 tier1.sources.source1.channels = channel1 tier1.channels.channel1.type = memory tier1.sinks.sink1.type = hdfs tier1.sinks.sink1.channel = channel1 tier1.sinks.sink1.hdfs.path = hdfs://192.168.4.444:8020/user/hadoop/flumelogs/ tier1.sinks.sink1.hdfs.fileType = DataStream tier1.sinks.sink1.hdfs.writeFormat= Text tier1.sinks.sink1.hdfs.batchSize = 1000 tier1.sinks.sink1.hdfs.rollCount = 0 tier1.sinks.sink1.hdfs.rollSize = 1000000 tier1.sinks.sink1.hdfs.rollInterval = 0 tier1.channels.channel1.capacity = 10000 tier1.channels.channel1.transactioncapacity=10000   Please Help.   ... View more