Reply
New Contributor
Posts: 4
Registered: ‎12-08-2017

FLUME - Rules to Extract Events from Log Files

[ Edited ]

I am looking to setup FLUME monitoring alerts in my cluster. Can anyone provide a set of examples to setup "Rules to Extract Events from Log Files"

 

Looking for typical Flume exception types as we are going live in production soon with new flume setup but without any monitoring in place for the flume agents. 

 

 

Contributor
Posts: 51
Registered: ‎11-03-2016

Hi,   Logs will be coming from multiple hosts or only one?

Hi,

 

Logs will be coming from multiple hosts or only one?

New Contributor
Posts: 4
Registered: ‎12-08-2017

Re: Hi, Logs will be coming from multiple hosts or only one?

[ Edited ]

Logs will be coming in 3 hosts

Highlighted
Contributor
Posts: 51
Registered: ‎11-03-2016

Re: Hi, Logs will be coming from multiple hosts or only one?

Hi,

 

 

You can use the following rules:

 

 

#Sources, channels, and sinks are defined per

# agent name, in this case 'agent'.

agent.sources  = source1

agent.channels = channel1

agent.sinks    = sink1

 

# For each source, channel, and sink, set

# standard properties.

agent.sources.source1.type     = seq

agent.sources.source1.channels = channel1

agent.sources.source.deserializer=org.apache.flume.sink.solr.morphline.BlobDeserializer$Builder

  

agent.channels.channel1.type   = file

agent.channels.channel1.checkpointDir = /myspool/flume1/checkpoint

agent.channels.channel1.dataDirs = /myspool/flume1/data

 

agent.sinks.loggerSink.type = logger

 

agent.sinks.sink1.type         = hdfs

agent.sinks.sink1.channel      = channel1

agent.sinks.sink1.hdfs.path    = /user/flumedata

agent.sinks.sink1.hdfs.filePrefix = SysLog-

agent.sinks.sink1.hdfs.fileType = DataStream

 

Go to the Flume folder and run the script

bin/flume-ng agent -n agent1 -c conf -f conf/flume.conf -Dflume.root.logger=DEBUG,console

Announcements