I am new to Metron. I have few log samples and I need to bring those logs to Metron. I want to understand how can I build a parser based on those logs. Where can I find a document about how to create a Grok or Java parser and how to can I use those in Metron?
Would appreciate the help.
Hi @R Patel
You can follow this guide to add new telemetry Data Source and adapt it to your logs :
I hope this helped
Hi @R Patel, Sebastien provided one of my favorite three documents on this topic. The other two are:
Similar to the previous citation, it is an end-to-end example using Squid logs as the example data, and creating a Grok parser to parse them.
Provides much more detail about how the parser fits into the overall Metron infrastructure, especially configuration, and a little guidance about Java parsers.
In a future release, Parsers will be a plug-in, with a maven archtype to help create new ones (Apache Jira METRON-777). So keep an eye out for that.
Also you may want to join the Apache Metron user or dev mailing lists. See http://metron.apache.org/community/
@R Patel , responding to your follow-up question about adding a Java parser:
When METRON-777 is added to Metron, which I expect will be within this week, there will be a true plug-in architecture for new parsers, along with a Maven Archetype and documentation for creating new parsers. Because it's coming very soon and will completely change the answer to your question, I encourage you to wait for it.
However, for the sake of completeness, here is a brief answer for the current state of the Metron code:
In the absence of a true plugin architecture, you simply have to add your new parser to the other Metron parsers.
1. Start by using git to clone the Metron codebase from https://github.com/apache/metron/
2. In your favorite IDE, navigate to metron/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ , and you will see that each of the existing parsers is a self-contained java package. Add a new package titled with the name of your parser.
3. The `bro` parser is a fairly simple example to follow. It demonstrates that your parser should extend `BasicParser` and provide implementations of its abstract methods.
4. However, the `bro` parser has a null configure method. Working in depth with Metron's configuration capability is too complex to include as part of this answer, but the `csv` parser provides a simple configuration method, and the `snort` processor a more sophisticated (but still follow-able) example of configuration. The configure() method is invoked automatically by the infrastructure at instance startup, and provides the opportunity for the parser instance to read its config.
5. By "your json file" I assume you mean the configuration. In general, rather than placing it in the code base, you would add it to the Zookeeper configuration for Metron, under the PARSER ConfigurationType, by using the same configuration tools used to add your custom config to the grok parser, but under the "sensor name" of your new parser.
That should be enough to give you a start. However, I reiterate that by the time you get well into such an implementation, METRON-777 should be available and you would want to switch gears to the plugin model. Hope this helps.
Thanks for sharing the details.
I have gone through it. In fact, I have worked on the Squid proxy logs and got it through.
In this case, the Grok parser was already there. I want to understand if I have a custom log source and I want to use a Java parser, Where do I need to place my parser (.java) file and what should be my .json file. If there is any document, It would be helpful for me.
Also, to follow on to Matt's answer:
If you add your parser this way, then you will have to maintain your own fork of metron, and manage syncing it with apache/metron/master or release.
Also, I think it is only implied by Matt, so I will explicitly point out that you will need to build the metron product yourself, and then do installs with YOUR built version of metron to get your parser installed. If not, you will have to do manual steps, one of which will include 'replace the metron-parsers-XXXX.jar on ALL the metron cluster nodes with yours'.
METRON-777 provides the foundation for the extension system, and moves the existing parsers over to it. It also introduces the maven archetype.
The follow on METRON-942 however provides the REST api support for installing and uninstalling parsers that you create. After 942 lands, your use case will be:
* install the mvn archetype
* create a directory
* run the archetype
* re-write the sample parser or replace it ( and the provided unit and integration test/configuration )
* build it
* go to the rest page and install it
* create a new kafka topic ( rest again )
* create a new storm topology for it
To the above reply, I have a query.
I want to dynamically populate a field, thus writing a custom java parser and configuring every parser's json file with the field value. I am populating the field in configure method. After deploying the parser the custom field doesn't comes up.
Can you please help in configuring, where I could be wrong. Also can you please mention the steps to be followed to create a custom field and populating a dynamic field.
Kindly reply back with suitable advise.
Thanks in advance.