Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to temporarily bypass Metron enrichments

avatar
Explorer

I am running an 8 node physical Metron cluster (2 search ES nodes, and a 6 node HDP 2.4 cluster that also runs the additional Metron services).

I started pushing bro logs onto the bro topic and have noticed it taking extremely long times to process (roughly 50 minutes from send to hitting the enrichment bolts). I'd like to speed this up and, at least for the short term, could live with bypassing the enrichment topic. Is there a simple way to do this?

1 ACCEPTED SOLUTION

avatar
Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
3 REPLIES 3

avatar
Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Explorer

I did this and it worked - I just wanted to clarify for others that when you make the backup of your .json file, it needs to be in a different directory than under /usr/metron/0.2.0BETA/config/zookeeper/. I made a backup by simply copying bro.json to bro.json.bkp and when I ran the PUSH command, I ended up with both a "bro" and a "bro.json" znode. I assume this is because the script gets all files in those dirs and removes the last extension ("bro.json" becomes "bro", and "bro.json.bkp" becomes "bro.json"), then uses that as the name on create.

avatar
Explorer

@Neha Sinha's solution worked, however that way it still goes through the enrichment tier, it just goes kafkaSpout -> Split -> Join -> Split -> Join -> Output. There is another way to do this that completely skips the enrichment tier and writes from the parser's topic (bro) directly to indexing via:

	  "parserConfig": {
            "kafka.topic" : "indexing"
          }

This exists in `/usr/metron/0.2.0BETA/config/zookeeper/parsers/bro.json` and can be pushed via `

/usr/metron/0.2.0BETA/bin/zk_load_configs.sh -z $zk -m PUSH -i /usr/metron/0.2.0BETA/config/zookeeper/`