FLUME - Rules to Extract Events from Log Files

[#1]

I am looking to setup FLUME monitoring alerts in my cluster. Can anyone provide a set of examples to setup "Rules to Extract Events from Log Files"

Looking for typical Flume exception types as we are going live in production soon with new flume setup but without any monitoring in place for the flume agents. 

[#2]

Hi,

Logs will be coming from multiple hosts or only one?

[#3]

Logs will be coming in 3 hosts

[#4]

Hi,

You can use the following rules:

#Sources, channels, and sinks are defined per

# agent name, in this case 'agent'.

agent.sources  = source1

agent.channels = channel1

agent.sinks    = sink1

# For each source, channel, and sink, set

# standard properties.

agent.sources.source1.type     = seq

agent.sources.source1.channels = channel1

agent.sources.source.deserializer=org.apache.flume.sink.solr.morphline.BlobDeserializer$Builder

agent.channels.channel1.type   = file

agent.channels.channel1.checkpointDir = /myspool/flume1/checkpoint

agent.channels.channel1.dataDirs = /myspool/flume1/data

agent.sinks.loggerSink.type = logger

agent.sinks.sink1.type         = hdfs

agent.sinks.sink1.channel      = channel1

agent.sinks.sink1.hdfs.path    = /user/flumedata

agent.sinks.sink1.hdfs.filePrefix = SysLog-

agent.sinks.sink1.hdfs.fileType = DataStream

Go to the Flume folder and run the script

bin/flume-ng agent -n agent1 -c conf -f conf/flume.conf -Dflume.root.logger=DEBUG,console