New Contributor
Posts: 3
Registered: ‎07-27-2016

having problem with flume interceptor search and replace

I am trying to alter the log event by using interceptor but it doesn't replace anything.


sample log event: - - [11/Jul/2016:06:29:48 +0000] "POST /moves/notify/new_measurements_available/ HTTP/1.1" 200 4426 "-" "Moves API"


here is the conf file


#Flume Configuration Starts
# Define a file channel called fileChannel on agent1

# Naming the components on the current agent
agent1.sinks = hdfs-sink1_1
agent1.sources = source1_1
agent1.channels = fileChannel1_1

# Define a channels for agent1

agent1.channels.fileChannel1_1.type = file
agent1.channels.fileChannel1_1.capacity = 200000
agent1.channels.fileChannel1_1.transactionCapacity = 1000

# Define a source for agent1

agent1.sources.source1_1.type = spooldir
agent1.sources.source1_1.spoolDir = /home/hduser/Desktop/SpoolingData

# Define an interceptor to alter data before writing to HDFS

agent1.sources.source1_1.interceptors = search-replace = search_replace = ^([\da-z:.]+) = done

agent1.sources.source1_1.fileHeader = false
agent1.sources.source1_1.fileSuffix = .COMPLETED

# Define a sink for agent1 (Sink is /flume_import under hdfs)

agent1.sinks.hdfs-sink1_1.type = hdfs
agent1.sinks.hdfs-sink1_1.hdfs.path = hdfs://localhost:9000/flume_sink
agent1.sinks.hdfs-sink1_1.hdfs.fileType = DataStream
agent1.sinks.hdfs-sink1_1.hdfs.batchSize = 1000
agent1.sinks.hdfs-sink1_1.hdfs.rollSize = 0
agent1.sinks.hdfs-sink1_1.hdfs.rollInterval =30
agent1.sinks.hdfs-sink1_1.hdfs.rollCount = 10000

# Binding the source and sink to the channel
agent1.sources.source1_1.channels = fileChannel1_1 = fileChannel1_1



but if I try 'searchPattern =' it works.


Secondly, what if I want to replace two string for example ' and +0000'. so do i need to use two interceptors or single interceptor can do this.

thanks in advance



New Contributor
Posts: 3
Registered: ‎07-27-2016

Re: having problem with flume interceptor search and replace

Solved the first problem, I had some mistake in regex. correct one is "^[A-Za-z0-9.: ]{11}+". can be modified but its working.
need help for second question.