How to temporarily bypass Metron enrichments

[#1]

I am running an 8 node physical Metron cluster (2 search ES nodes, and a 6 node HDP 2.4 cluster that also runs the additional Metron services).

I started pushing bro logs onto the bro topic and have noticed it taking extremely long times to process (roughly 50 minutes from send to hitting the enrichment bolts). I'd like to speed this up and, at least for the short term, could live with bypassing the enrichment topic. Is there a simple way to do this?

[#2 · Accepted solution]

@Jon Zeolla

Hi,

You can bypass enrichment for bro by following the below steps :-

1.Establish SSH connection to the node where metron service(s) is installed.

2.On the specified node navigate to the following path :- /usr/metron/0.2.0BETA/config/zookeeper/enrichments

3.Create a backup of the existing bro.json file in some directory so that you can use it later.

4.Now edit the bro.json file and change the content to the following :-

{
  "index": "bro",
  "batchSize": 5
}

5.Now execute the following command to push the config to the zookeeper :-

/usr/metron/0.2.0BETA/bin/zk_load_configs.sh -z ZOOKEEPER_HOST:2181 -m PUSH -i /usr/metron/0.2.0BETA/config/zookeeper/

Please replace the ZOOKEEPER_HOST text with your zookeeper node hostname.

6.Perform the following command to make sure the config got pushed all good to zookeeper and no errors are seen.

/usr/metron/0.2.0BETA/bin/zk_load_configs.sh -z ZOOKEEPER_HOST:2181 -m DUMP

Since the current bro.json file @/usr/metron/0.2.0BETA/config/zookeeper/enrichments/bro.json does not contain any mention of enrichment so enrichment should be by passed for the the Bro logs.

Please let me know if the above steps solves the issue you are facing.

Regards,

neha

[#3]

I did this and it worked - I just wanted to clarify for others that when you make the backup of your .json file, it needs to be in a different directory than under /usr/metron/0.2.0BETA/config/zookeeper/. I made a backup by simply copying bro.json to bro.json.bkp and when I ran the PUSH command, I ended up with both a "bro" and a "bro.json" znode. I assume this is because the script gets all files in those dirs and removes the last extension ("bro.json" becomes "bro", and "bro.json.bkp" becomes "bro.json"), then uses that as the name on create.

[#4]

@Neha Sinha's solution worked, however that way it still goes through the enrichment tier, it just goes kafkaSpout -> Split -> Join -> Split -> Join -> Output. There is another way to do this that completely skips the enrichment tier and writes from the parser's topic (bro) directly to indexing via:

	  "parserConfig": {
            "kafka.topic" : "indexing"
          }

This exists in `/usr/metron/0.2.0BETA/config/zookeeper/parsers/bro.json` and can be pushed via `

/usr/metron/0.2.0BETA/bin/zk_load_configs.sh -z $zk -m PUSH -i /usr/metron/0.2.0BETA/config/zookeeper/`