Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Flume: Reformat syslog message

Flume: Reformat syslog message

New Contributor



I'm building the following setup for my central logging infrastructure:

rSyslog Client ==> Flume Syslog Source ==> Memory Channel ==> Elastic Search Sink ==> ES Cluister <== Kibana 3 Web UI


Unfortunately some vendors do not provide well formatted syslog messages. In my case the date/time is some kind of weird:


2013:09:17-09:03:03 ulogd[30168]: id="2001" severity="info" sys="SecureNet" foo="bar" ...


I would like to use the Morphline interceptor to modify the date/time to a valid format and save it to the corresponding headers. So i use a simple "readLine" and "gork" to get out my fields (year, month, day, ...) as described in the manual/examples. Thats the easy part.


But now I'm get stuck on how i can put the single fields together again:


  1. The single fields of the date/time should be converted to a timestamp and overwrite the existing header (@fields.timestamp)
  2. The timestamp should be converted to an RFC3339 format and overwrite the timestamp header of the flume syslog source (@timestamp)
    Don't know if this is possible. Maybe i should use NetCat Source instead and parse the whole message myself?
  3. The wrong date/time should be removed from the message because it's not needed there anymore


Thank you very much for any help



Re: Flume: Reformat syslog message

New Contributor

OK, havent read thre reference exactly enough.


I should be able to use the existing date/time in the message field as an input for "convertTimestamp" to get a well formated RFC3339 timestamp. So, point 2 is partialy solve.


How can i convert it to an unix teimstamp then?




Re: Flume: Reformat syslog message

Expert Contributor
convertTimestamp has an option to convert to unix time. See