Support Questions

Find answers, ask questions, and share your expertise
Celebrating as our community reaches 100,000 members! Thank you!

Interceptor not working in flume


Im trying to movie log files from server X to hdfs. while moving i want to filter or replace some data, so for that i'm using search&replace interceptor. But it not working Please Help


Here My config file


Local Agent:

agent.sources = localsource
agent.channels = memoryChannel
agent.sinks = avro_Sink


agent.sources.localsource.type = spooldir = /bin/bash -c
agent.sources.localsource.spoolDir = /home/dwh/teja/Flumedata/
agent.sources.localsource.fileHeader = true

# The channel can be defined as follows.
agent.sources.localsource.channels = memoryChannel

# Each sink's type must be defined
agent.sinks.avro_Sink.type = avro
agent.sinks.avro_Sink.port= 8021
agent.sinks.avro_Sink.avro.batchSize = 100
agent.sinks.avro_Sink.avro.rollCount = 0
agent.sinks.avro_Sink.avro.rollSize = 65000000
agent.sinks.avro_Sink.avro.rollInterval = 0

agent.sources.localsource.interceptors = search-replace = search_replace

# Remove leading alphanumeric characters in an event body. = '/^\s*\#+|\#+$/' = '|'
#Specify the channel the sink should use = memoryChannel

# Each channel's type is defined.
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.capacity = 10000
agent.channels.memoryChannel.transactionCapacity = 1000


Here My Hdfs Agent:

tier1.sources = source1
tier1.channels = channel1
tier1.sinks = sink1

# For each source, channel, and sink, set
# standard properties.
tier1.sources.source1.type = avro
tier1.sources.source1.bind =
tier1.sources.source1.channels = channel1
tier1.channels.channel1.type = memory
tier1.sinks.sink1.type = hdfs = channel1
tier1.sinks.sink1.hdfs.path = hdfs://
tier1.sinks.sink1.hdfs.fileType = DataStream
tier1.sinks.sink1.hdfs.writeFormat= Text
tier1.sinks.sink1.hdfs.batchSize = 100
tier1.sinks.sink1.hdfs.rollCount = 0
tier1.sinks.sink1.hdfs.rollSize = 65000000
tier1.sinks.sink1.hdfs.rollInterval = 0
tier1.channels.channel1.capacity = 10000


Please say any thing need to add

Thanks in advance




Is this the pattern that you are trying to match on: ^\s*\#+|\#+$


Try it without quoting or the forward slashes, like: = ^\s*\#+|\#+$ = |



View solution in original post



Is this the pattern that you are trying to match on: ^\s*\#+|\#+$


Try it without quoting or the forward slashes, like: = ^\s*\#+|\#+$ = |



Hi, Thanks for Your reply

It's working fine i got it Before but forgot to update my answer. As You said need to remove single quotes and slashes then it's working fine.
I used directly means, Instead of this ^\s*\#+|\#+$ I used direct ## to replace with pipe line symbol.