Support Questions

Find answers, ask questions, and share your expertise

Nifi to ingest syslog and convert raw message to JSON and parse message to JSON

avatar
New Contributor

Please help me to ingest the following syslog message in NIFI. (The syslog parse failed as it is unable to parse the syslog) I want to convert the log into the following JSON format and even store the raw message in JSON

I want the parsing to be dynamic for example the key value should be automatically determined with the = delimeter and not static( where to write regex to for each and every attribute) .

Log Message :

2017:11:20-21:11:53 firewall01 ulogd[25916]: id="2002" severity="info" sys="SecureNet" sub="packetfilter" name="Packet accepted" action="accept" fwrule="5" initf="ppp0" outitf="eth1" mark="0x3518" app="1304" srcmac="00:15:5d:65:1f:06" srcip="10.242.3.2" dstip="10.90.90.104" proto="6" length="52" tos="0x00" prec="0x00" ttl="127" srcport="7554" dstport="8080" tcpflags="SYN"

The Output : ( Manually created JSON ).

{

'"_time " : "2017:11:20-21:11:53 " ,

"hostname":"firewall01 ",

"logtype": " ulogd[25916]",

"id":"2002",

"severity":"info",

"sys":"SecureNet",

"sub":"packetfilter",

"name":"Packet accepted",

"action":"accept",

"fwrule":"5",

"initf":"pp0",

"outif":"eth1",

"mark":"0x3518",

"app":"1304",

"srcmac":"00:15:5d:65:1f:06",

"srcip":"10.242.3.2",

"dstip":"10.90.90.104",

"proto":"6",

"length":"52",

"tos":"0x00",

"prec":"0x00",
"ttl":"127",

"srcport":"7554",

"dstport":"8080",

"tcpflags":"SYN",

"_raw" : "2017:11:20-21:11:53 firewall01 ulogd[25916]: id="2002" severity="info" sys="SecureNet" sub="packetfilter" name="Packet accepted" action="accept" fwrule="5" initf="ppp0" outitf="eth1" mark="0x3518" app="1304" srcmac="00:15:5d:65:1f:06" srcip="10.242.3.2" dstip="10.90.90.104" proto="6" length="52" tos="0x00" prec="0x00" ttl="127" srcport="7554" dstport="8080" tcpflags="SYN" "

}

9 REPLIES 9

avatar
Explorer

@Prithiviraj Krishnakumar

I have a problem like your problem.

Did you solve it?

would you share it?

avatar
Contributor

Hi, I also have the same problem, I am able to get the 'body' into valid json but want the key value pairs (separated by '=') into json, having no luck?

avatar
Super Guru

@Griggsy ,

 

Instead of using the ParseSyslog processor, try using ParseSyslog5424, which has a closer implementation to the RFC 5424 standard.

 

When you use ParseSyslog5424, the attributes in the structured data part of of the syslog message are parsed and added to the flowfiles as attributes. For example, the following syslog message:

 

 

<35>1 2013-10-11T22:14:15.003Z client_machine su - - [SDID@0 utilization="high" os="linux"] 'su root' failed for joe on /dev/pts/2

 

 

 Will yield a flowfile including the following attributes:

araujo_0-1646781582719.png

 

Cheers,

André

--
Was your question answered? Please take some time to click on "Accept as Solution" below this post.
If you find a reply useful, say thanks by clicking on the thumbs up button.
--
Was your question answered? Please take some time to click on "Accept as Solution" below this post.
If you find a reply useful, say thanks by clicking on the thumbs up button.

avatar
Contributor

@araujo 

 

Thanks André

 

What if I wanted to turn the key value pairs separated by '=' in json content as in the original question? for instance the following syslog:

 

Griggsy_0-1646904682271.png

converted into:

[ {

"sig" : "0",

"arch" : "c000003e",

"syscall" : "87"

}]

 

I'm aware this can be done using regex to create attributes and then attributeToJson but some of my logs have hundreds of key value pairs so that's not an option, there must be a way to convert it using record processing? i.e convertRecord

avatar
Super Guru

Hi @Griggsy ,

 

You don't need to use regex for this. You can, for example, connect the output of the ParseSyslog5424 directly to the following AttributesToJson processor:

araujo_1-1646912142875.png

The output of this will be a JSON like the one below:

{
  "syslog.structuredData.SDID@0.os" : "linux",
  "syslog.structuredData.SDID@0.utilization" : "high"
}

You can then use a JoltTransformJSON process to transform the above into your final product. For example, the following JOLT specification:

[
  {
    "operation": "shift",
    "spec": {
      "syslog.structuredData.*.*": {
        "@": "&(1,2)"
      }
    }
  }
]

 

Will produce the following output:

{
  "os" : "linux",
  "utilization" : "high"
}

 

This is the flow:

araujo_2-1646912636306.png

 

Cheers,

André

--
Was your question answered? Please take some time to click on "Accept as Solution" below this post.
If you find a reply useful, say thanks by clicking on the thumbs up button.

--
Was your question answered? Please take some time to click on "Accept as Solution" below this post.
If you find a reply useful, say thanks by clicking on the thumbs up button.

avatar
Contributor

@araujo Thanks very useful indeed!

avatar
Community Manager

@Griggsy, can you please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.  



Regards,

Vidya Sargur,
Community Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:

avatar
Contributor

Hi @VidyaSargur 

 

I do not see an option for 'Accept as Solution' below the post, I assume because I didn't ask the original question.

 

Regards