I'm using a GROK parser, consuming from kafka queue 'auditd'. The STORM topology for the 'auditd' parser looks like it's working correctly:
We can see 9540 messages 'Acked', with only a handful of reported errors in the logs (for records which didn't match my GROK expression).
We can also see within Metron UI that it's reporting throughput:
However, this data never makes it to the enrichment or indexing topology. As we can see the STORM topology's have never received any data:
For some reason the data is not being passed properly down stream. FWIW I used the Metron UI to configure this telemetry source. I can also see json files defined for the predefined telemetry sources in:
/usr/metron/0.4.0/config/zookeeper/{indexing,parsers,enrichments}
However in these folders I do not see any configuration files for the new auditd source. I have assumed that this is not required because the REST API configures the topology via UI.
Is there anything obviously missing that might explain why these messages are not being passed upstream?
Thanks in advance
If you have anything similar like that:
Check the detailed logs of the relevant topology workers. The Metron Storm workers will run on the hosts where your Storm Supervisors run.
On that host(s), the logs are usually at :
/var/log/storm/workers-artifacts/bro-2-1496396293/6700/worker.log (or something along those lines depending on what parser etc.). Find your relevant 'bro-2-1496396293' topology id via the Storm UI on port 8744.
If you only find a file ' worker.yaml ' at /var/log/storm/workers-artifacts/<topology_ID>/6700/ it means that your worker was not able to start up in the first place. Then the problem is deeper and you need to find the cause in
/var/log/storm/supervisor.log (on that same host)
Alternatively:
If set up correctly, via the Storm UI you can sometimes drill all the way through to the Spout / Bolt which seem not doing anything, and check the logs web based. This can be done at 'files' at the level where you get the details of the "Executors (All time)"
:
