Reply
New Contributor
Posts: 2
Registered: ‎09-17-2013

Flume: Reformat syslog message

Hi,

 

I'm building the following setup for my central logging infrastructure:

rSyslog Client ==> Flume Syslog Source ==> Memory Channel ==> Elastic Search Sink ==> ES Cluister <== Kibana 3 Web UI

 

Unfortunately some vendors do not provide well formatted syslog messages. In my case the date/time is some kind of weird:

 

2013:09:17-09:03:03 ulogd[30168]: id="2001" severity="info" sys="SecureNet" foo="bar" ...

 

I would like to use the Morphline interceptor to modify the date/time to a valid format and save it to the corresponding headers. So i use a simple "readLine" and "gork" to get out my fields (year, month, day, ...) as described in the manual/examples. Thats the easy part.

 

But now I'm get stuck on how i can put the single fields together again:

 

  1. The single fields of the date/time should be converted to a timestamp and overwrite the existing header (@fields.timestamp)
  2. The timestamp should be converted to an RFC3339 format and overwrite the timestamp header of the flume syslog source (@timestamp)
    Don't know if this is possible. Maybe i should use NetCat Source instead and parse the whole message myself?
  3. The wrong date/time should be removed from the message because it's not needed there anymore

 

Thank you very much for any help

Urs

New Contributor
Posts: 2
Registered: ‎09-17-2013

Re: Flume: Reformat syslog message

OK, havent read thre reference exactly enough.

 

I should be able to use the existing date/time in the message field as an input for "convertTimestamp" to get a well formated RFC3339 timestamp. So, point 2 is partialy solve.

 

How can i convert it to an unix teimstamp then?

 

 

Highlighted
Cloudera Employee
Posts: 146
Registered: ‎08-21-2013

Re: Flume: Reformat syslog message

convertTimestamp has an option to convert to unix time. Seehttp://cloudera.github.io/cdk/docs/current/cdk-morphlines/morphlinesReferenceGuide.html#convertTimestamp

Wolfgang.