Grok Parsing Error for New Metron Telemetry Data Source

[#1]

Hi,

I'm getting the following error from Storm's ParserBolt in a single node deployment of Metron 0.3.1:

java.lang.IllegalStateException: Grok parser Error: Grok statement produced a null message. Original message was: {"hostname":"myhost","timestamp":"1010101010","cpu_user":"8.522706","cpu_sys":"29.210946","cpu_nice":"0.000000","cpu_idle":"62.266348","cpu_wait":"0.000000","cpu_irq":"0.000000","cpu_soft_irq":"0.000000","cpu_stolen":"0.000000","load_one":"4.945312","load_five":"4.391113","load_fifteen":"4.385254"} and the parsed message was: {} . Check the pattern at: /apps/metron/patterns/my_telemetry_source on {"hostname":"myhost","timestamp":"1010101010","cpu_user":"8.522706","cpu_sys":"29.210946","cpu_nice":"0.000000","cpu_idle":"62.266348","cpu_wait":"0.000000","cpu_irq":"0.000000","cpu_soft_irq":"0.000000","cpu_stolen":"0.000000","load_one":"4.945312","load_five":"4.391113","load_fifteen":"4.385254"} at org.apache.metron.parsers.GrokParser.parse(GrokParser.java:174) at org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:45) at org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:132) at org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734) at org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466) at org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40) at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) at org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853) at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: Grok statement produced a null message. Original message was: {"hostname":"myhost","timestamp":"1010101010","cpu_user":"8.522706","cpu_sys":"29.210946","cpu_nice":"0.000000","cpu_idle":"62.266348","cpu_wait":"0.000000","cpu_irq":"0.000000","cpu_soft_irq":"0.000000","cpu_stolen":"0.000000","load_one":"4.945312","load_five":"4.391113","load_fifteen":"4.385254"} and the parsed message was: {} . Check the pattern at: /apps/metron/patterns/my_telemetry_source at org.apache.metron.parsers.GrokParser.parse(GrokParser.java:152) ... 12 more

My input lines are of the following form:

{"hostname":"myhost","timestamp":"1010101010","cpu_user":"8.522706","cpu_sys":"29.210946","cpu_nice":"0.000000","cpu_idle":"62.266348","cpu_wait":"0.000000","cpu_irq":"0.000000","cpu_soft_irq":"0.000000","cpu_stolen":"0.000000","load_one":"4.945312","load_five":"4.391113","load_fifteen":"4.385254"}
{"hostname":"myhost","timestamp":"1012101010","cpu_user":"8.522706","cpu_sys":"29.210946","cpu_nice":"0.000000","cpu_idle":"62.266348","cpu_wait":"0.000000","cpu_irq":"0.000000","cpu_soft_irq":"0.000000","cpu_stolen":"0.000000","load_one":"4.945312","load_five":"4.391113","load_fifteen":"4.385254"}

My Grok Pattern is:

MY_TELEMETRY_SOURCE_DELIMITED %{GREEDYDATA:UNWANTED}{"hostname":"%{HOSTNAME:hostname}","timestamp":"%{NUMBER:timestamp}","cpu_user":"%{NUMBER:cpu_user}","cpu_sys":"%{NUMBER:cpu_sys}","cpu_nice":"%{NUMBER:cpu_nice}","cpu_idle":"%{NUMBER:cpu_idle}","cpu_wait":"%{NUMBER:cpu_wait}","cpu_irq":"%{NUMBER:cpu_irq}","cpu_soft_irq":"%{NUMBER:cpu_soft_irq}","cpu_stolen":"%{NUMBER:cpu_stolen}","load_one":"%{NUMBER:load_one}","load_five":"%{NUMBER:load_five}","load_fifteen":"%{NUMBER:load_fifteen}"}%{GREEDYDATA:UNWANTED}

Which according to the GrokConstructor (http://grokconstructor.appspot.com) matches correctly.

All Metron services are up and the pattern is staged at /apps/metron/patterns/my_telemetry_source as evident by the error log.

My parser configuration for the parsing topology is located at /usr/metron/0.3.1/config/zookeeper/parsers/my_telemetry_source.json and consists of the following:

{
    "parserClassName": "org.apache.metron.parsers.GrokParser",
    "sensorTopic": "my_telemetry_source",
    "parserConfig": {
        "grokPath": "/apps/metron/patterns/my_telemetry_source",        
        "patternLabel": "MY_TELEMETRY_SOURCE_DELIMITED",
        "timestampField": "timestamp"
    }
}

I have tried this setup both with and without applying the patternLabel, MY_TELEMETRY_SOURCE_DELIMITED, getting the same error message in both cases.

I appreciate your help.

[#2]

Hello @H S,

Have you find a solution for Grok parsing Error :

java.lang.IllegalStateException: Grok parser Error: Grok statement produced a null message. Original message was: {"hostname":"my.machine","timestamp":"1490746284"} and the parsed message was: {}

I actually have the same error with a very simple Grok pattern :

PARSE_TXT %{DATA:email}\:%{DATA:name}

For the moment, i just try to parse the following text :

toto:toto
tata:tata
titi:titi

I have tested this on the grok constructor website: http://grokconstructor.appspot.com/ and it correctly match.

Anyone already had this error ?

My parser configuration :

{
 "parserClassName": "org.apache.metron.parsers.GrokParser",
 "sensorTopic": "parse_txt",
 "parserConfig": {
 "grokPath": "/apps/metron/patterns/parse_txt",
 "patternLabel": "PARSE_TXT"
 	}
}

Thanks !

[#3]

@Sebastien Lehuede:

No, I didn't find the root cause. But in my case, it was fairly easy to use the CSV parser instead of the Grok parser, with slight modifications to my telemetry source. You can find more details about the CSV parser here:

https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html

Hope this helps.

[#4]

Hi @H S

Thanks for your answer, i have increased resources on my Full Dev VM and used Metron UI to create new sensor, currently it works.

But i have a new problem, Elasticsearch not index logs. I don't know if logs correctly reach Elasticsearch because in Storm UI, in parserBolt -> Output stats (for my topic), i don't have anythings. (Screen Below).

Has anyone ever had a problem with Kafka Stream ? Where logs correctly pass thought NiFi, correctly receive by Kafka and parse by Grok but are not redirect to Elasticsearch.

bolt-logs.png

[#5]

I'm having a similar issue. In my case I'm feeding Yaf data to Kafka (it works correctly). Then it gives the parsing error:

java.lang.IllegalStateException: Grok parser Error: Grok statement produced a null message. Original message was:

I used the Metron's default YAF Grok pattern. I have checked the logs in http://grokconstructor.appspot.com and it is able to correctly match the pattern. Furthermore, I use the "Test" button on Metron's UI to check whether the patter is able to extract the fields from the log and it also works correctly.

Does anyone knows why this fails?

Example Log:

2018-04-04 10:05:32.533|2018-04-04 10:05:32.533| 0.000| 0.000| 6| 10.5.0.83| 2869| 10.5.0.97|52023| AR| 0| 0| 0|c019c454|00000000|000|000| 1| 40| 0| 0| 0|

Yaf Pattern

YAF_TIME_FORMAT %{YEAR:UNWANTED}-%{MONTHNUM:UNWANTED}-%{MONTHDAY:UNWANTED}[T ]%{HOUR:UNWANTED}:%{MINUTE:UNWANTED}:%{SECOND:UNWANTED} 

YAF_DELIMITED %{YAF_TIME_FORMAT:start_time}\|%{YAF_TIME_FORMAT:end_time}\|%{SPACE:UNWANTED}%{BASE10NUM:duration}\|%{SPACE:UNWANTED}%{BASE10NUM:rtt}\|%{SPACE:UNWANTED}%{INT:protocol}\|%{SPACE:UNWANTED}%{IP:ip_src_addr}\|%{SPACE:UNWANTED}%{INT:ip_src_port}\|%{SPACE:UNWANTED}%{IP:ip_dst_addr}\|%{SPACE:UNWANTED}%{INT:ip_dst_port}\|%{SPACE:UNWANTED}%{DATA:iflags}\|%{SPACE:UNWANTED}%{DATA:uflags}\|%{SPACE:UNWANTED}%{DATA:riflags}\|%{SPACE:UNWANTED}%{DATA:ruflags}\|%{SPACE:UNWANTED}%{WORD:isn}\|%{SPACE:UNWANTED}%{DATA:risn}\|%{SPACE:UNWANTED}%{DATA:tag}\|%{GREEDYDATA:rtag}\|%{SPACE:UNWANTED}%{INT:pkt}\|%{SPACE:UNWANTED}%{INT:oct}\|%{SPACE:UNWANTED}%{INT:rpkt}\|%{SPACE:UNWANTED}%{INT:roct}\|%{SPACE:UNWANTED}%{INT:app}\|%{GREEDYDATA:end_reason}