Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Flume: Reformat syslog message

Highlighted

Flume: Reformat syslog message

New Contributor

Hi,

 

I'm building the following setup for my central logging infrastructure:

rSyslog Client ==> Flume Syslog Source ==> Memory Channel ==> Elastic Search Sink ==> ES Cluister <== Kibana 3 Web UI

 

Unfortunately some vendors do not provide well formatted syslog messages. In my case the date/time is some kind of weird:

 

2013:09:17-09:03:03 ulogd[30168]: id="2001" severity="info" sys="SecureNet" foo="bar" ...

 

I would like to use the Morphline interceptor to modify the date/time to a valid format and save it to the corresponding headers. So i use a simple "readLine" and "gork" to get out my fields (year, month, day, ...) as described in the manual/examples. Thats the easy part.

 

But now I'm get stuck on how i can put the single fields together again:

 

  1. The single fields of the date/time should be converted to a timestamp and overwrite the existing header (@fields.timestamp)
  2. The timestamp should be converted to an RFC3339 format and overwrite the timestamp header of the flume syslog source (@timestamp)
    Don't know if this is possible. Maybe i should use NetCat Source instead and parse the whole message myself?
  3. The wrong date/time should be removed from the message because it's not needed there anymore

 

Thank you very much for any help

Urs

2 REPLIES 2

Re: Flume: Reformat syslog message

New Contributor

OK, havent read thre reference exactly enough.

 

I should be able to use the existing date/time in the message field as an input for "convertTimestamp" to get a well formated RFC3339 timestamp. So, point 2 is partialy solve.

 

How can i convert it to an unix teimstamp then?

 

 

Re: Flume: Reformat syslog message

Expert Contributor
convertTimestamp has an option to convert to unix time. Seehttp://cloudera.github.io/cdk/docs/current/cdk-morphlines/morphlinesReferenceGuide.html#convertTimestamp

Wolfgang.