question Re: Metron pcap analysis vs wireshark in Archives of Support Questions (Read Only)

Metron pcap analysis vs wireshark

nnat25191 — Thu, 07 Sep 2017 02:00:15 GMT

Hello all,

I’m new to Metron and i’ve been reading this forum of questions, explanatory, and guides. However, I’m still missing a few pieces of puzzles for my understanding of Metron’s capability. Any feedback is appreciated.

Someone mentioned that for fine grain analysis of PCAP data we would need to use Wireshark. Wireshark is great, but it’s slow specially when we want to load a few PCAP files in and correlate them at once. Also, I would like to correlate/analyze my data from all types that were ingested into Metron. What’s the limitation of PCAP analysis in Metron?Why do I need to use Wireshark when I already have all my data in Metron?
Can I manually add PCAP files to Metron? My sample scenario would be to have Metron installed and setup locally and periodically load in data collected from another Metron cluster after a network security testing or security vulnerability assessment.

Again, thank you for all feedbacks in advance.

Re: Metron pcap analysis vs wireshark

jsirota — Sat, 28 Oct 2017 02:36:44 GMT

You use Metron to capture, stream, and store PCAP on HDFS. You do so at massive scale and massive volume, which is not something that Wireshark can handle. You can then use Metron services to search and filter PCAP from HDFS, and once filtered save it (once it's at a size that Wireshark can consume) and load it in Wireshark for further examination. You need several components to accomplish this functionality. The overview and the docs are located here:

https://github.com/apache/metron/tree/master/metron-platform/metron-pcap-backend

Re: Metron pcap analysis vs wireshark

cstella — Sat, 28 Oct 2017 02:57:10 GMT

So, as @jsirota said, Metron focuses on storage and capture at scale. Another dimension to the problem is filtering that which you've stored. In addition to storm topologies and DPDK-based tools for capture, we can also filter the stored raw pcap data filtering via a few dimensions:

time
destination port
destination address
source port
source address
binary regex for content

This search happens over all of your data in parallel and the output is a pcap file of just the matching packets. This is suitable for further analysis in wireshark to bring in other layer metadata.

To recap, we are mostly a capture to HDFS and then filter for targeted correlation later.

Re: Metron pcap analysis vs wireshark

nnat25191 — Tue, 31 Oct 2017 01:40:57 GMT

Thank you @cstella and @jsirota

The pcap data stored in HDFS is sequence files. How do you view them in Wireshark? My guess would be somehow get the pcap_inspector service to spit out the result of the filter in PCAP format?

Thank you.

Re: Metron pcap analysis vs wireshark

cstella — Tue, 31 Oct 2017 01:55:32 GMT

Actually pcap_inspector.sh is just for inspecting the pcap files and dumping header information from the packets in the sequence file. What you probably want is pcap_query, which will execute a query over the sequence files and write out a pcap suitable for opening with wireshark.

Re: Metron pcap analysis vs wireshark

nallen — Tue, 31 Oct 2017 03:08:35 GMT

> The pcap data stored in HDFS is sequence files. How do you view them in Wireshark? My guess would be somehow get the pcap_inspector service to spit out the result of the filter in PCAP format?

@Arian Trayen As @cstella mentioned, "pcap_query" does exactly that. It will output a libpcap-compliant file that you can open with Wireshark.