Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.

having problem with flume interceptor search and replace

Highlighted

having problem with flume interceptor search and replace

New Contributor

I am trying to alter the log event by using interceptor but it doesn't replace anything.

 

sample log event:


w2e.fi:443 54.284.11.111 - - [11/Jul/2016:06:29:48 +0000] "POST /moves/notify/new_measurements_available/ HTTP/1.1" 200 4426 "-" "Moves API"

 

here is the conf file

 

#Flume Configuration Starts
# Define a file channel called fileChannel on agent1

# Naming the components on the current agent
agent1.sinks = hdfs-sink1_1
agent1.sources = source1_1
agent1.channels = fileChannel1_1

# Define a channels for agent1

agent1.channels.fileChannel1_1.type = file
agent1.channels.fileChannel1_1.capacity = 200000
agent1.channels.fileChannel1_1.transactionCapacity = 1000

# Define a source for agent1

agent1.sources.source1_1.type = spooldir
agent1.sources.source1_1.spoolDir = /home/hduser/Desktop/SpoolingData

# Define an interceptor to alter data before writing to HDFS

agent1.sources.source1_1.interceptors = search-replace
agent1.sources.source1_1.interceptors.search-replace.type = search_replace
agent1.sources.source1_1.interceptors.search-replace.searchPattern = ^([\da-z:.]+) 
agent1.sources.source1_1.interceptors.search-replace.replaceString = done

agent1.sources.source1_1.fileHeader = false
agent1.sources.source1_1.fileSuffix = .COMPLETED

# Define a sink for agent1 (Sink is /flume_import under hdfs)

agent1.sinks.hdfs-sink1_1.type = hdfs
agent1.sinks.hdfs-sink1_1.hdfs.path = hdfs://localhost:9000/flume_sink
agent1.sinks.hdfs-sink1_1.hdfs.fileType = DataStream
agent1.sinks.hdfs-sink1_1.hdfs.writeFormat=Text
agent1.sinks.hdfs-sink1_1.hdfs.batchSize = 1000
agent1.sinks.hdfs-sink1_1.hdfs.rollSize = 0
agent1.sinks.hdfs-sink1_1.hdfs.rollInterval =30
agent1.sinks.hdfs-sink1_1.hdfs.rollCount = 10000

# Binding the source and sink to the channel
agent1.sources.source1_1.channels = fileChannel1_1
agent1.sinks.hdfs-sink1_1.channel = fileChannel1_1

 

 

but if I try 'searchPattern = w2e.fi:443' it works.

 

Secondly, what if I want to replace two string for example 'w2e.fi:443 and +0000'. so do i need to use two interceptors or single interceptor can do this.


thanks in advance