Reply
Highlighted
New Contributor
Posts: 3
Registered: ‎07-27-2016

having problem with flume interceptor search and replace

I am trying to alter the log event by using interceptor but it doesn't replace anything.

 

sample log event:


w2e.fi:443 54.284.11.111 - - [11/Jul/2016:06:29:48 +0000] "POST /moves/notify/new_measurements_available/ HTTP/1.1" 200 4426 "-" "Moves API"

 

here is the conf file

 

#Flume Configuration Starts
# Define a file channel called fileChannel on agent1

# Naming the components on the current agent
agent1.sinks = hdfs-sink1_1
agent1.sources = source1_1
agent1.channels = fileChannel1_1

# Define a channels for agent1

agent1.channels.fileChannel1_1.type = file
agent1.channels.fileChannel1_1.capacity = 200000
agent1.channels.fileChannel1_1.transactionCapacity = 1000

# Define a source for agent1

agent1.sources.source1_1.type = spooldir
agent1.sources.source1_1.spoolDir = /home/hduser/Desktop/SpoolingData

# Define an interceptor to alter data before writing to HDFS

agent1.sources.source1_1.interceptors = search-replace
agent1.sources.source1_1.interceptors.search-replace.type = search_replace
agent1.sources.source1_1.interceptors.search-replace.searchPattern = ^([\da-z:.]+) 
agent1.sources.source1_1.interceptors.search-replace.replaceString = done

agent1.sources.source1_1.fileHeader = false
agent1.sources.source1_1.fileSuffix = .COMPLETED

# Define a sink for agent1 (Sink is /flume_import under hdfs)

agent1.sinks.hdfs-sink1_1.type = hdfs
agent1.sinks.hdfs-sink1_1.hdfs.path = hdfs://localhost:9000/flume_sink
agent1.sinks.hdfs-sink1_1.hdfs.fileType = DataStream
agent1.sinks.hdfs-sink1_1.hdfs.writeFormat=Text
agent1.sinks.hdfs-sink1_1.hdfs.batchSize = 1000
agent1.sinks.hdfs-sink1_1.hdfs.rollSize = 0
agent1.sinks.hdfs-sink1_1.hdfs.rollInterval =30
agent1.sinks.hdfs-sink1_1.hdfs.rollCount = 10000

# Binding the source and sink to the channel
agent1.sources.source1_1.channels = fileChannel1_1
agent1.sinks.hdfs-sink1_1.channel = fileChannel1_1

 

 

but if I try 'searchPattern = w2e.fi:443' it works.

 

Secondly, what if I want to replace two string for example 'w2e.fi:443 and +0000'. so do i need to use two interceptors or single interceptor can do this.


thanks in advance

 

 

New Contributor
Posts: 3
Registered: ‎07-27-2016

Re: having problem with flume interceptor search and replace

Solved the first problem, I had some mistake in regex. correct one is "^[A-Za-z0-9.: ]{11}+". can be modified but its working.
need help for second question.

Announcements

Our community is getting a little larger. And a lot better.


Learn More about the Cloudera and Hortonworks community merger planned for late July and early August.