Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Basic questions about apache metron

Basic questions about apache metron

New Contributor

We are planning to build and offer fully managed/outsourced SOC for our clients. We plan do go with Apache Metron since we like open source SW and have very low budget.

I have some very basic questions, I would greatly appreciate if someone could take at look at them:

  1. Is there a way to implement multi tenant apache Metron? In the beginning we will have very few clients, but it would be nice to have a single dashboard that would monitor all clients. If this is not possible, its not a big problem.
  2. Is it possible to capture network traffic in a way that we are not in between the internet and the client network? We would like to collect the data in the minimally intrusive way, so even if all our servers fail, their operations remain intact. Maybe configure a switch to mirror all traffic to Metron server? How would I implement this on Metron server?
  3. What would be the best architecture for our use case:
    - Have just one Metron server placed in client’s premises and our SOC analyst would connect to the web interface
    - Or just have some Kafka server in client’s premises and have a VPS with Metron for each client in the cloud.
    - Or please propose the the best architecture for our use case
  4. How to collect data that is not traffic related? Lets say we want to parse logs from all client servers, client anti-viruses, etc. What should we install on those servers to get the data to metron? I see that apache Nifi is retired, so we would like to avoid that. So the question is: is there a “metron sensor app” that we will install on all servers and those will feed the logs to metron?

Thank you

3 REPLIES 3

Re: Basic questions about apache metron

Guru

Hi, good to hear you're looking at Metron. It certainly makes a lot of sense in an MSSP context. To answer your questions...

Yes, Metron works in a multi-tenant environment. The solution to this usually relies on the meta data functions in Metron parsers, which allow you to tag ownership of data in the keys sent to Metron's kafka topics. There is more info in the docs here: https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html#Metadata

The other side of multi-tenancy usually involves using the Ranger component of HDP to ensure either permissions based, or full cryptographic separation of tenant's data using TDE.

When handling network traffic capture, the best method is to use Metron's Fastcapa probe against a span/mirror port from a switch. This means you can capture traffic from any point in a network, as long as you can span it on the switches. This can be used to capture internal traffic as well as internal to external traffic. You would usually want to dedicate a network card to this, and then setup the fastcapa sensor to run on that interface.

In terms of the architecture for collection of PCAP a lot of that depends on the anticipated volume. If you have a huge amount of capture, often you will need a Metron cluster co-located. However, for relatively modest volumes, keeping a local kafka, potentially mirrored to the core Metron Kafka elsewhere might be possible, bandwidth allowing. There are other more sophisticated methods possible involving things like minifi based probes, but that gets a little involved for this answer!

Metron is also often used to collect non-network sources, logs are the main one, particularly server, IDS, firewall, and AV engine logs. From the Metron perspective these just need to get into Kafka. To do this, the most common solution is to use NiFi, which is very much alive and well, and the best choice for this sort of transport. I have seem a number of successful log transport solutions based on NiFi feeding into Metron, and the Hortonworks team, among others in the community are actively working on NiFi and MiNiFi based sensor apps for Metron which deploy to sites, and ship data back into a core Metron cluster.

Re: Basic questions about apache metron

New Contributor

Thank you very much for your excellent reply.

Couple of days ago, there was a big warning at the top of the NiFi page, that the project is retired. It is not there any more.

The whole apache metron looks complicated and fragile, so I hope we will be able to do it.

Re: Basic questions about apache metron

New Contributor

@Zsolt Szabo, Hi, Could you please share your experience while deploying Apache Metron in Multitenant mode ? We are beginning to build same setup and would be great to know any learning from your work.

Don't have an account?
Coming from Hortonworks? Activate your account here