Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

having problem with flume interceptor search and replace

having problem with flume interceptor search and replace

New Contributor

I am trying to alter the log event by using interceptor but it doesn't replace anything.


sample log event: - - [11/Jul/2016:06:29:48 +0000] "POST /moves/notify/new_measurements_available/ HTTP/1.1" 200 4426 "-" "Moves API"


here is the conf file


#Flume Configuration Starts
# Define a file channel called fileChannel on agent1

# Naming the components on the current agent
agent1.sinks = hdfs-sink1_1
agent1.sources = source1_1
agent1.channels = fileChannel1_1

# Define a channels for agent1

agent1.channels.fileChannel1_1.type = file
agent1.channels.fileChannel1_1.capacity = 200000
agent1.channels.fileChannel1_1.transactionCapacity = 1000

# Define a source for agent1

agent1.sources.source1_1.type = spooldir
agent1.sources.source1_1.spoolDir = /home/hduser/Desktop/SpoolingData

# Define an interceptor to alter data before writing to HDFS

agent1.sources.source1_1.interceptors = search-replace = search_replace = ^([\da-z:.]+) = done

agent1.sources.source1_1.fileHeader = false
agent1.sources.source1_1.fileSuffix = .COMPLETED

# Define a sink for agent1 (Sink is /flume_import under hdfs)

agent1.sinks.hdfs-sink1_1.type = hdfs
agent1.sinks.hdfs-sink1_1.hdfs.path = hdfs://localhost:9000/flume_sink
agent1.sinks.hdfs-sink1_1.hdfs.fileType = DataStream
agent1.sinks.hdfs-sink1_1.hdfs.batchSize = 1000
agent1.sinks.hdfs-sink1_1.hdfs.rollSize = 0
agent1.sinks.hdfs-sink1_1.hdfs.rollInterval =30
agent1.sinks.hdfs-sink1_1.hdfs.rollCount = 10000

# Binding the source and sink to the channel
agent1.sources.source1_1.channels = fileChannel1_1 = fileChannel1_1



but if I try 'searchPattern =' it works.


Secondly, what if I want to replace two string for example ' and +0000'. so do i need to use two interceptors or single interceptor can do this.

thanks in advance




Re: having problem with flume interceptor search and replace

New Contributor

Solved the first problem, I had some mistake in regex. correct one is "^[A-Za-z0-9.: ]{11}+". can be modified but its working.
need help for second question.