Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Metron pcap analysis vs wireshark

Solved Go to solution
Highlighted

Metron pcap analysis vs wireshark

Contributor

Hello all,

I’m new to Metron and i’ve been reading this forum of questions, explanatory, and guides. However, I’m still missing a few pieces of puzzles for my understanding of Metron’s capability. Any feedback is appreciated.

  1. Someone mentioned that for fine grain analysis of PCAP data we would need to use Wireshark. Wireshark is great, but it’s slow specially when we want to load a few PCAP files in and correlate them at once. Also, I would like to correlate/analyze my data from all types that were ingested into Metron. What’s the limitation of PCAP analysis in Metron?Why do I need to use Wireshark when I already have all my data in Metron?
  2. Can I manually add PCAP files to Metron? My sample scenario would be to have Metron installed and setup locally and periodically load in data collected from another Metron cluster after a network security testing or security vulnerability assessment.

Again, thank you for all feedbacks in advance.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Metron pcap analysis vs wireshark

Explorer

So, as @jsirota said, Metron focuses on storage and capture at scale. Another dimension to the problem is filtering that which you've stored. In addition to storm topologies and DPDK-based tools for capture, we can also filter the stored raw pcap data filtering via a few dimensions:

  1. time
  2. destination port
  3. destination address
  4. source port
  5. source address
  6. binary regex for content

This search happens over all of your data in parallel and the output is a pcap file of just the matching packets. This is suitable for further analysis in wireshark to bring in other layer metadata.

To recap, we are mostly a capture to HDFS and then filter for targeted correlation later.

View solution in original post

5 REPLIES 5
Highlighted

Re: Metron pcap analysis vs wireshark

Explorer

You use Metron to capture, stream, and store PCAP on HDFS. You do so at massive scale and massive volume, which is not something that Wireshark can handle. You can then use Metron services to search and filter PCAP from HDFS, and once filtered save it (once it's at a size that Wireshark can consume) and load it in Wireshark for further examination. You need several components to accomplish this functionality. The overview and the docs are located here:

https://github.com/apache/metron/tree/master/metron-platform/metron-pcap-backend

Re: Metron pcap analysis vs wireshark

Explorer

So, as @jsirota said, Metron focuses on storage and capture at scale. Another dimension to the problem is filtering that which you've stored. In addition to storm topologies and DPDK-based tools for capture, we can also filter the stored raw pcap data filtering via a few dimensions:

  1. time
  2. destination port
  3. destination address
  4. source port
  5. source address
  6. binary regex for content

This search happens over all of your data in parallel and the output is a pcap file of just the matching packets. This is suitable for further analysis in wireshark to bring in other layer metadata.

To recap, we are mostly a capture to HDFS and then filter for targeted correlation later.

View solution in original post

Highlighted

Re: Metron pcap analysis vs wireshark

Contributor

Thank you @cstella and @jsirota

The pcap data stored in HDFS is sequence files. How do you view them in Wireshark? My guess would be somehow get the pcap_inspector service to spit out the result of the filter in PCAP format?

Thank you.

Highlighted

Re: Metron pcap analysis vs wireshark

Explorer

Actually pcap_inspector.sh is just for inspecting the pcap files and dumping header information from the packets in the sequence file. What you probably want is pcap_query, which will execute a query over the sequence files and write out a pcap suitable for opening with wireshark.

Highlighted

Re: Metron pcap analysis vs wireshark

Rising Star

> The pcap data stored in HDFS is sequence files. How do you view them in Wireshark? My guess would be somehow get the pcap_inspector service to spit out the result of the filter in PCAP format?

@Arian Trayen As @cstella mentioned, "pcap_query" does exactly that. It will output a libpcap-compliant file that you can open with Wireshark.

Don't have an account?
Coming from Hortonworks? Activate your account here