Grok Parsing Error for New Metron Telemetry Data Source

[#1]

Hello @H S,

Have you find a solution for Grok parsing Error :

java.lang.IllegalStateException: Grok parser Error: Grok statement produced a null message. Original message was: {"hostname":"my.machine","timestamp":"1490746284"} and the parsed message was: {}

I actually have the same error with a very simple Grok pattern :

PARSE_TXT %{DATA:email}\:%{DATA:name}

For the moment, i just try to parse the following text :

toto:toto
tata:tata
titi:titi

I have tested this on the grok constructor website: http://grokconstructor.appspot.com/ and it correctly match.

Anyone already had this error ?

My parser configuration :

{
 "parserClassName": "org.apache.metron.parsers.GrokParser",
 "sensorTopic": "parse_txt",
 "parserConfig": {
 "grokPath": "/apps/metron/patterns/parse_txt",
 "patternLabel": "PARSE_TXT"
 	}
}

Thanks !

[#2]

@Sebastien Lehuede:

No, I didn't find the root cause. But in my case, it was fairly easy to use the CSV parser instead of the Grok parser, with slight modifications to my telemetry source. You can find more details about the CSV parser here:

https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html

Hope this helps.

[#3]

Hi @H S

Thanks for your answer, i have increased resources on my Full Dev VM and used Metron UI to create new sensor, currently it works.

But i have a new problem, Elasticsearch not index logs. I don't know if logs correctly reach Elasticsearch because in Storm UI, in parserBolt -> Output stats (for my topic), i don't have anythings. (Screen Below).

Has anyone ever had a problem with Kafka Stream ? Where logs correctly pass thought NiFi, correctly receive by Kafka and parse by Grok but are not redirect to Elasticsearch.

bolt-logs.png

[#4]

I'm having a similar issue. In my case I'm feeding Yaf data to Kafka (it works correctly). Then it gives the parsing error:

java.lang.IllegalStateException: Grok parser Error: Grok statement produced a null message. Original message was:

I used the Metron's default YAF Grok pattern. I have checked the logs in http://grokconstructor.appspot.com and it is able to correctly match the pattern. Furthermore, I use the "Test" button on Metron's UI to check whether the patter is able to extract the fields from the log and it also works correctly.

Does anyone knows why this fails?

Example Log:

2018-04-04 10:05:32.533|2018-04-04 10:05:32.533| 0.000| 0.000| 6| 10.5.0.83| 2869| 10.5.0.97|52023| AR| 0| 0| 0|c019c454|00000000|000|000| 1| 40| 0| 0| 0|

Yaf Pattern

YAF_TIME_FORMAT %{YEAR:UNWANTED}-%{MONTHNUM:UNWANTED}-%{MONTHDAY:UNWANTED}[T ]%{HOUR:UNWANTED}:%{MINUTE:UNWANTED}:%{SECOND:UNWANTED} 

YAF_DELIMITED %{YAF_TIME_FORMAT:start_time}\|%{YAF_TIME_FORMAT:end_time}\|%{SPACE:UNWANTED}%{BASE10NUM:duration}\|%{SPACE:UNWANTED}%{BASE10NUM:rtt}\|%{SPACE:UNWANTED}%{INT:protocol}\|%{SPACE:UNWANTED}%{IP:ip_src_addr}\|%{SPACE:UNWANTED}%{INT:ip_src_port}\|%{SPACE:UNWANTED}%{IP:ip_dst_addr}\|%{SPACE:UNWANTED}%{INT:ip_dst_port}\|%{SPACE:UNWANTED}%{DATA:iflags}\|%{SPACE:UNWANTED}%{DATA:uflags}\|%{SPACE:UNWANTED}%{DATA:riflags}\|%{SPACE:UNWANTED}%{DATA:ruflags}\|%{SPACE:UNWANTED}%{WORD:isn}\|%{SPACE:UNWANTED}%{DATA:risn}\|%{SPACE:UNWANTED}%{DATA:tag}\|%{GREEDYDATA:rtag}\|%{SPACE:UNWANTED}%{INT:pkt}\|%{SPACE:UNWANTED}%{INT:oct}\|%{SPACE:UNWANTED}%{INT:rpkt}\|%{SPACE:UNWANTED}%{INT:roct}\|%{SPACE:UNWANTED}%{INT:app}\|%{GREEDYDATA:end_reason}