Support Questions
Find answers, ask questions, and share your expertise

How to drop full record in log file by searching a keyword in interceptor using flume

Solved Go to solution
Highlighted

How to drop full record in log file by searching a keyword in interceptor using flume

Contributor

Hi, Guys

 

           I want to drop full record by searching a key using interceptor in flume is that possible to drop full record in flume?

  For example in my log i have record like : (this record in pagenotfound.php) by searching pagenotfound.php  this keyword i want to drop that full record is that possible ?

 

Please Help

Thanks in advance

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: How to drop full record in log file by searching a keyword in interceptor using flume

Contributor
Hi, Ben

Now interceptor working fine. I changed above code like this

agent.sources.localsource.interceptors = search-replace regex-filter
agent.sources.localsource.interceptors.search-replace.type = search_replace

# Remove leading alphanumeric characters in an event body.
agent.sources.localsource.interceptors.search-replace.searchPattern = ###|##
agent.sources.localsource.interceptors.search-replace.replaceString = |

#agent.sources.localsource.interceptors = regex-filter
agent.sources.localsource.interceptors.regex-filter.type = regex_filter

# Remove full event body.
agent.sources.localsource.interceptors.regex-filter.regex = .*PageInsource\:pagenotfound.php.*
agent.sources.localsource.interceptors.regex-filter.excludeEvents = true
Now its working pretty fine.

View solution in original post

8 REPLIES 8
Highlighted

Re: How to drop full record in log file by searching a keyword in interceptor using flume

Expert Contributor

Perhaps the Regex Filtering Interceptor is what you were looking for?

 

https://flume.apache.org/FlumeUserGuide.html#regex-filtering-interceptor

Highlighted

Re: How to drop full record in log file by searching a keyword in interceptor using flume

Contributor
Hi Ben thanks for your answer. As You said i can use Regex Filtering Interceptor, But i already using Search and Replace Interceptor for one filter can i use both in one agent at same time
Highlighted

Re: How to drop full record in log file by searching a keyword in interceptor using flume

Expert Contributor

It would appear you can "chain" them by putting the interceptors that are desired in a list in the order you want them applied. I have never personally done it, so I can't say for sure. Hope this helps!

 

[1]"Flume supports chaining of interceptors. This is made possible through by specifying the list of interceptor builder class names in the configuration. Interceptors are specified as a whitespace separated list in the source configuration. The order in which the interceptors are specified is the order in which they are invoked. The list of events returned by one interceptor is passed to the next interceptor in the chain."

 

 

[1]https://flume.apache.org/FlumeUserGuide.html#flume-interceptors

Highlighted

Re: How to drop full record in log file by searching a keyword in interceptor using flume

Contributor

Hi Ben

 

          Thanks a lot for replying, As you said i read that but i didn't get this thing

This is made possible through by specifying the list of interceptor builder class names in the configuration

as you said i want to define interceptors names like interceptor 1 and interceptor 2 this what you saying right.

Highlighted

Re: How to drop full record in log file by searching a keyword in interceptor using flume

Expert Contributor

guessing like this: (assuming you want search-replace to be applied first, regex second)

 

....

agent.sources.localsource.interceptors = search-replace regex

agent.sources.localsource.interceptors.search-replace.type = search_replace

agent.sources.localsource.interceptors.regex.type = regex_filter

....

 

 

 

 

Highlighted

Re: How to drop full record in log file by searching a keyword in interceptor using flume

Contributor
hi Ben

I tried like this as you said

agent.sources.localsource.interceptors = search-replace
agent.sources.localsource.interceptors.search-replace.type = search_replace

# Remove leading alphanumeric characters in an event body.
agent.sources.localsource.interceptors.search-replace.searchPattern = ###|##
agent.sources.localsource.interceptors.search-replace.replaceString = |

agent.sources.localsource.interceptors = regex-filter
agent.sources.localsource.interceptors.regex-filter.type = regex_filter

# Remove full event body.
agent.sources.localsource.interceptors.regex-filter.searchPattern = "pagenotfound.php"
agent.sources.localsource.interceptors.regex-filter.excludeEvents = true

But flume not writing events at all if i remove regex_filter interceptor then its writing events so you have any idea how to use it .

Thanks in advance.
Highlighted

Re: How to drop full record in log file by searching a keyword in interceptor using flume

Contributor
Hi, Ben

Now interceptor working fine. I changed above code like this

agent.sources.localsource.interceptors = search-replace regex-filter
agent.sources.localsource.interceptors.search-replace.type = search_replace

# Remove leading alphanumeric characters in an event body.
agent.sources.localsource.interceptors.search-replace.searchPattern = ###|##
agent.sources.localsource.interceptors.search-replace.replaceString = |

#agent.sources.localsource.interceptors = regex-filter
agent.sources.localsource.interceptors.regex-filter.type = regex_filter

# Remove full event body.
agent.sources.localsource.interceptors.regex-filter.regex = .*PageInsource\:pagenotfound.php.*
agent.sources.localsource.interceptors.regex-filter.excludeEvents = true
Now its working pretty fine.

View solution in original post

Highlighted

Re: How to drop full record in log file by searching a keyword in interceptor using flume

Expert Contributor

Glad to Hear it!