Support Questions
Find answers, ask questions, and share your expertise

listensyslog help

Expert Contributor

I am listening syslog messages using listen syslog.

the format of syslog message I am receiving is

(<PRIORITY>)(version)(TIMESTAMP) (HOSTNAME) (BODY)

Body contains different sets of information as source address, destination address,uid,uname...up to 200 sets

When I use listensyslog processor, every message is going in to invalid. I am not sure why it is going in to invalid.

Also, as the messages are going in to invalid, I am getting flowfile as stream of messages (other than expected as one message per flow file). As every flow file has multiple (100s) of messages, it is becoming tedious to figure out messages and process.

I am thinking to store every set as a column in hive using puthiveql.

ex:

sourceaddress destinationaddress uid uname
1.1.1.1 1.1.1.2 a1 David
1.1.1.1 1.1.1.2 a2 Michelle

listensyslog properties

42736-listen.png

8 REPLIES 8

Expert Contributor
@Shu

can you help here please.

You'll need to provide an example message that is going to invalid so we can figure out why, it is most likely encountering something in of the sections before the body that doesn't technically conform to the syslog RFC.

Expert Contributor

@Bryan Bende

sorry for delay

Below is an example

<134>1 2017-06-30T14:05:21.000000+02:00 anc-anc307-txt 208 [set complete="<134>1 2017-06-30T14:05:21+02:00 anc-anc307 txt_RPC 1712 - [file@1 position=\"newposition\" size=\"6984061\" name=\"C:\\\\Program Files\\\\Microsoft\\\\log.LOG\"\] position=\"newposition\" size=\"6984061\" name=\"C:\\\\Program Files\\\\Microsoft\\\\log.LOG\"][meta sequenceId=\"123456\"\]

I created a simple flow with GenerateFlowFile -> ParseSyslog -> UpdateAttribute

The GenerateFlowFile has the Custom Text property set to the exact message you provided above, and ParseSyslog uses the same parser as ListenSyslog.

With this test, the above message is parsed successfully:

47390-parsesyslog.png

Expert Contributor
@Bryan Bende

Thanks for the above approach, could you please let me know the properties you have used in parsesyslog please.

Super Guru
@Mark

There are no properties that we need to setup in parsesyslog processor, if your Syslog message in accordance to RFC5424 and RFC3164 formats and adds attributes to the FlowFile for each of the parts of the Syslog message.

Just drag and drop the processor to canvas then processor can parse the log file and writes the below attributes to the flowfile.

Write Attributes:-

NameDescription
syslog.priorityThe priority of the Syslog message.
syslog.severityThe severity of the Syslog message derived from the priority.
syslog.facilityThe facility of the Syslog message derived from the priority.
syslog.versionThe optional version from the Syslog message.
syslog.timestampThe timestamp of the Syslog message.
syslog.hostnameThe hostname or IP address of the Syslog message.
syslog.senderThe hostname of the Syslog server that sent the message.
syslog.bodyThe body of the Syslog message, everything after the hostname.

Example:-

GenerateFlowfile configs:-

45799-generateflowfile.png

as we kept Custom Text property as your log mentioned in comment.

Parse SysLog:-

drag and drop the processor then processor parses the log file and writes the attributes as shown in the below screenshot.

Output:-

45800-syslog-output.png

Expert Contributor

@Shu @Bryan Bende

Hi Both,

Thank you for your support.

A quick question:

There is a small deviation in the message which I mentioned.

I am getting length of the message before the message

(lengt of message)(<PRIORITY>)(version)(TIMESTAMP) (HOSTNAME) (BODY)

ex:

362 <134>1 2017-06-30T14:05:21.000000+02:00 anc-anc307-txt 208 [set complete="<134>1 2017-06-30T14:05:21+02:00 anc-anc307 txt_RPC 1712 - [file@1 position=\"newposition\" size=\"6984061\" name=\"C:\\\\Program Files\\\\Microsoft\\\\log.LOG\"\] position=\"newposition\" size=\"6984061\" name=\"C:\\\\Program Files\\\\Microsoft\\\\log.LOG\"][meta sequenceId=\"123456\"\]

When I parse the above message my parsesyslog is sending to failure. But getting length before the whole message is according to rfc format only, then why my parsesyslog is sending the message to failure.

please help me to understand

Can you link to an RFC that shows that the length at the beginning is a valid format?

I haven't seen that before and I can say that NiFi is definitely not setup to handle that, but if that is truly part of an RFC then we should create a JIRA to fix that.