Interceptor not working in flume

Tejaponnaluru — Fri, 16 Sep 2022 10:12:39 GMT

Im trying to movie log files from server X to hdfs. while moving i want to filter or replace some data, so for that i'm using search&replace interceptor. But it not working Please Help

Here My config file

Local Agent:

agent.sources = localsource
agent.channels = memoryChannel
agent.sinks = avro_Sink

agent.sources.localsource.type = spooldir
#agent.sources.localsource.shell = /bin/bash -c
agent.sources.localsource.spoolDir = /home/dwh/teja/Flumedata/
agent.sources.localsource.fileHeader = true

# The channel can be defined as follows.
agent.sources.localsource.channels = memoryChannel

# Each sink's type must be defined
agent.sinks.avro_Sink.type = avro
agent.sinks.avro_Sink.hostname=192.168.4.444
agent.sinks.avro_Sink.port= 8021
agent.sinks.avro_Sink.avro.batchSize = 100
agent.sinks.avro_Sink.avro.rollCount = 0
agent.sinks.avro_Sink.avro.rollSize = 65000000
agent.sinks.avro_Sink.avro.rollInterval = 0

agent.sources.localsource.interceptors = search-replace
agent.sources.localsource.interceptors.search-replace.type = search_replace

# Remove leading alphanumeric characters in an event body.
agent.sources.localsource.interceptors.search-replace.searchPattern = '/^\s*\#+|\#+$/'
agent.sources.localsource.interceptors.search-replace.replaceString = '|'
#Specify the channel the sink should use
agent.sinks.avro_Sink.channel = memoryChannel

# Each channel's type is defined.
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.capacity = 10000
agent.channels.memoryChannel.transactionCapacity = 1000

Here My Hdfs Agent:

tier1.sources = source1
tier1.channels = channel1
tier1.sinks = sink1

# For each source, channel, and sink, set
# standard properties.
tier1.sources.source1.type = avro
tier1.sources.source1.bind = 192.168.4.51
tier1.sources.source1.port=8021
tier1.sources.source1.channels = channel1
tier1.channels.channel1.type = memory
tier1.sinks.sink1.type = hdfs
tier1.sinks.sink1.channel = channel1
tier1.sinks.sink1.hdfs.path = hdfs://192.168.4.51:8020/user/hadoop/flumelogs/
tier1.sinks.sink1.hdfs.fileType = DataStream
tier1.sinks.sink1.hdfs.writeFormat= Text
tier1.sinks.sink1.hdfs.batchSize = 100
tier1.sinks.sink1.hdfs.rollCount = 0
tier1.sinks.sink1.hdfs.rollSize = 65000000
tier1.sinks.sink1.hdfs.rollInterval = 0
tier1.channels.channel1.capacity = 10000
tier1.channels.channel1.transactioncapacity=1000

Please say any thing need to add

Thanks in advance

Re: Interceptor not working in flume

pdvorak — Tue, 12 Apr 2016 16:14:07 GMT

Is this the pattern that you are trying to match on: ^\s*\#+|\#+$

Try it without quoting or the forward slashes, like:

agent.sources.localsource.interceptors.search-replace.searchPattern = ^\s*\#+|\#+$
agent.sources.localsource.interceptors.search-replace.replaceString = |

-pd

Re: Interceptor not working in flume

Tejaponnaluru — Wed, 13 Apr 2016 04:08:43 GMT

Hi, Thanks for Your reply

It's working fine i got it Before but forgot to update my answer. As You said need to remove single quotes and slashes then it's working fine.
I used directly means, Instead of this ^\s*\#+|\#+$ I used direct ## to replace with pipe line symbol.

question Re: Interceptor not working in flume in Archives of Support Questions (Read Only)

Interceptor not working in flume

Re: Interceptor not working in flume

Re: Interceptor not working in flume