Support Questions

Find answers, ask questions, and share your expertise

Nifi: KV filter

avatar
New Contributor

I want to parse application log data having format: <date> key1="value1" key2="value2" .... and generate output as json.
And this log data does not follow any pattern, and these key value pairs are dynamic. So I am looking for something similar to KV filter in logstash, where I can get all the key values from log.
Not sure how to proceed with this. Appreciate a quick help here. 

4 REPLIES 4

avatar
Contributor

hello @spa 

I've been looking for this also but it doesn't exist. Then, you can use a script (python, groovy,...) In case you have performance issue with scriptprocessor, you can improve the situation using the trick here: 

InvokeScriptedProcessor template (a faster ExecuteScript)  

avatar
Explorer

Hello @mburgess 

Can you please answer the above question from VidyaSargur?

avatar
Master Guru

If you use GrokReader you can use the same kv filter from logstash: https://community.cloudera.com/t5/Support-Questions/Grok-Patterns-Expressions-for-capturing-comma-se...

 

avatar
Explorer

Hi @mburgess 

Can you please elaborate on which property needs to be configured in the GrokReader controller service for using the kv filter?

I'm trying to parse the incoming key=value pair.

Input: key1=value1,key2=value2,key3=value3,key4=value4
output: I need key1, key2, key3, key4 as attributes and their respective values as attribute values

I can use %{GREEDYDATA:msgbody} in the GrokExpression property but I do not know where to provide 
kv {
      source = "msgbody"
}

Your help is appreciated