We are planning to build and offer fully managed/outsourced SOC for our clients. We plan do go with Apache Metron since we like open source SW and have very low budget.
I have some very basic questions, I would greatly appreciate if someone could take at look at them:
Hi, good to hear you're looking at Metron. It certainly makes a lot of sense in an MSSP context. To answer your questions...
Yes, Metron works in a multi-tenant environment. The solution to this usually relies on the meta data functions in Metron parsers, which allow you to tag ownership of data in the keys sent to Metron's kafka topics. There is more info in the docs here: https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html#Metadata
The other side of multi-tenancy usually involves using the Ranger component of HDP to ensure either permissions based, or full cryptographic separation of tenant's data using TDE.
When handling network traffic capture, the best method is to use Metron's Fastcapa probe against a span/mirror port from a switch. This means you can capture traffic from any point in a network, as long as you can span it on the switches. This can be used to capture internal traffic as well as internal to external traffic. You would usually want to dedicate a network card to this, and then setup the fastcapa sensor to run on that interface.
In terms of the architecture for collection of PCAP a lot of that depends on the anticipated volume. If you have a huge amount of capture, often you will need a Metron cluster co-located. However, for relatively modest volumes, keeping a local kafka, potentially mirrored to the core Metron Kafka elsewhere might be possible, bandwidth allowing. There are other more sophisticated methods possible involving things like minifi based probes, but that gets a little involved for this answer!
Metron is also often used to collect non-network sources, logs are the main one, particularly server, IDS, firewall, and AV engine logs. From the Metron perspective these just need to get into Kafka. To do this, the most common solution is to use NiFi, which is very much alive and well, and the best choice for this sort of transport. I have seem a number of successful log transport solutions based on NiFi feeding into Metron, and the Hortonworks team, among others in the community are actively working on NiFi and MiNiFi based sensor apps for Metron which deploy to sites, and ship data back into a core Metron cluster.
Thank you very much for your excellent reply.
Couple of days ago, there was a big warning at the top of the NiFi page, that the project is retired. It is not there any more.
The whole apache metron looks complicated and fragile, so I hope we will be able to do it.
@Zsolt Szabo, Hi, Could you please share your experience while deploying Apache Metron in Multitenant mode ? We are beginning to build same setup and would be great to know any learning from your work.