Metron pcap analysis vs wireshark

[#1]

Hello all,

I’m new to Metron and i’ve been reading this forum of questions, explanatory, and guides. However, I’m still missing a few pieces of puzzles for my understanding of Metron’s capability. Any feedback is appreciated.

1. Someone mentioned that for fine grain analysis of PCAP data we would need to use Wireshark. Wireshark is great, but it’s slow specially when we want to load a few PCAP files in and correlate them at once. Also, I would like to correlate/analyze my data from all types that were ingested into Metron. What’s the limitation of PCAP analysis in Metron?Why do I need to use Wireshark when I already have all my data in Metron?
2. Can I manually add PCAP files to Metron? My sample scenario would be to have Metron installed and setup locally and periodically load in data collected from another Metron cluster after a network security testing or security vulnerability assessment.

Again, thank you for all feedbacks in advance.

[#2]

You use Metron to capture, stream, and store PCAP on HDFS. You do so at massive scale and massive volume, which is not something that Wireshark can handle. You can then use Metron services to search and filter PCAP from HDFS, and once filtered save it (once it's at a size that Wireshark can consume) and load it in Wireshark for further examination. You need several components to accomplish this functionality. The overview and the docs are located here:

https://github.com/apache/metron/tree/master/metron-platform/metron-pcap-backend

[#3 · Accepted solution]

So, as @jsirota said, Metron focuses on storage and capture at scale. Another dimension to the problem is filtering that which you've stored. In addition to storm topologies and DPDK-based tools for capture, we can also filter the stored raw pcap data filtering via a few dimensions:

1. time
2. destination port
3. destination address
4. source port
5. source address
6. binary regex for content

This search happens over all of your data in parallel and the output is a pcap file of just the matching packets. This is suitable for further analysis in wireshark to bring in other layer metadata.

To recap, we are mostly a capture to HDFS and then filter for targeted correlation later.

[#4]

Thank you @cstella and @jsirota

The pcap data stored in HDFS is sequence files. How do you view them in Wireshark? My guess would be somehow get the pcap_inspector service to spit out the result of the filter in PCAP format?

Thank you.

[#5]

Actually pcap_inspector.sh is just for inspecting the pcap files and dumping header information from the packets in the sequence file. What you probably want is pcap_query, which will execute a query over the sequence files and write out a pcap suitable for opening with wireshark.

[#6]

> The pcap data stored in HDFS is sequence files. How do you view them in Wireshark? My guess would be somehow get the pcap_inspector service to spit out the result of the filter in PCAP format?

@Arian Trayen As @cstella mentioned, "pcap_query" does exactly that. It will output a libpcap-compliant file that you can open with Wireshark.