Created 09-06-2017 07:00 PM
Hello all,
I’m new to Metron and i’ve been reading this forum of questions, explanatory, and guides. However, I’m still missing a few pieces of puzzles for my understanding of Metron’s capability. Any feedback is appreciated.
Again, thank you for all feedbacks in advance.
Created 10-27-2017 07:57 PM
So, as @jsirota said, Metron focuses on storage and capture at scale. Another dimension to the problem is filtering that which you've stored. In addition to storm topologies and DPDK-based tools for capture, we can also filter the stored raw pcap data filtering via a few dimensions:
This search happens over all of your data in parallel and the output is a pcap file of just the matching packets. This is suitable for further analysis in wireshark to bring in other layer metadata.
To recap, we are mostly a capture to HDFS and then filter for targeted correlation later.
Created 10-27-2017 07:36 PM
You use Metron to capture, stream, and store PCAP on HDFS. You do so at massive scale and massive volume, which is not something that Wireshark can handle. You can then use Metron services to search and filter PCAP from HDFS, and once filtered save it (once it's at a size that Wireshark can consume) and load it in Wireshark for further examination. You need several components to accomplish this functionality. The overview and the docs are located here:
https://github.com/apache/metron/tree/master/metron-platform/metron-pcap-backend
Created 10-27-2017 07:57 PM
So, as @jsirota said, Metron focuses on storage and capture at scale. Another dimension to the problem is filtering that which you've stored. In addition to storm topologies and DPDK-based tools for capture, we can also filter the stored raw pcap data filtering via a few dimensions:
This search happens over all of your data in parallel and the output is a pcap file of just the matching packets. This is suitable for further analysis in wireshark to bring in other layer metadata.
To recap, we are mostly a capture to HDFS and then filter for targeted correlation later.
Created 10-30-2017 06:40 PM
Created 10-30-2017 06:55 PM
Actually pcap_inspector.sh is just for inspecting the pcap files and dumping header information from the packets in the sequence file. What you probably want is pcap_query, which will execute a query over the sequence files and write out a pcap suitable for opening with wireshark.
Created 10-30-2017 08:08 PM
> The pcap data stored in HDFS is sequence files. How do you view them in Wireshark? My guess would be somehow get the pcap_inspector service to spit out the result of the filter in PCAP format?
@Arian Trayen As @cstella mentioned, "pcap_query" does exactly that. It will output a libpcap-compliant file that you can open with Wireshark.