Pcap data index to ElasticSearch

[#1]

I've installed and configured pcap service and replay for our apache metron. The pcap data has been captured and stored in HDFS /apps/metron/pcap. However, I could not find information of how to index these files into ElasticSearch and make it available on Apache Metron dashboard. I found a related thread, but didn't answer my question or I'm just confused? https://community.hortonworks.com/questions/36622/how-to-use-the-metron-ui-to-see-the-pcap-data.html

1. How to get the pcap data collected/stored in HDFS indexed into ElasticSearch?

2. How to get the pcap panel on Metron dashboard like the old version of Metron?

Any feedback is greatly appreciated.

[#2 · Accepted solution]

> 1. How to get the pcap data collected/stored in HDFS indexed into ElasticSearch?

In Apache Metron there is not a mechanism to ingest raw pcap data into Elasticsearch. I have found a search index like Elasticsearch more useful for higher level meta information like flows.

There is a tool called Pcap Query to search and retrieve slices of the raw pcap stored in HDFS. This queries against the data stored in HDFS and returns a libpcap-compliant file containing the raw pcap data that you can then load into 3rd party tools like Wireshark.

> 2. How to get the pcap panel on Metron dashboard like the old version of Metron?

The Pcap Panel from the original OpenSOC project was not carried forward due to technical limitations.

[#3]

Thank you so much for your quick response @nallen

1. Your explanation makes sense. Wireshark is very useful, however it's slow and I like the idea of collecting all pcap data and store it in one place, then make it searchable on a dashboard. Would it be possible to extract metadata fields from pcap files and index them into ElasticSearch with Metron?

2. What's the technical limitation?

thank you!

[#4]

(1) Would it be possible to extract metadata fields from pcap files and index them into ElasticSearch with Metron?

Yes, that is effectively what Metron does when it ingests Bro and YAF telemetry. We let those external tools, tools that are best-in-class at extracting metadata from raw pcap, do the extraction. Metron then consumes that metadata, enriches it, triages it, and indexes it in a search index like Elasticsearch. So your metadata ends up in Elasticsearch, which I think is your end goal here.

(2) What's the technical limitation?

The PCAP Panel was a custom extension of an old, forked version of Kibana, as I remember it. It was not something we were able to just carry forward without a major overhaul.

[#5]

Wonderful!

1. extending to the same question, there are pcap files in my HDFS, does this mean that the metadata fields are highly possibly made it into ElasticSearch and available? Do I need to create that dataflow/index or is it already in ElasticSearch by kafkaConsumer setup for pcap? How do I recognize pcap metadata in Elasticsearch indexes (only see yaf, snort, bro, and squid)?

2. there seems to be no pcap parser available in metron sensor rest ui. There are only parsers for bro, snort, yaf, websphere, asa, JsonMap, and Grok. What do you recommend for Pcap parser? I used JsonMap and I don't see error in the log, but I don't know if it parse correctly for metadata fields.

Again, thank you for your feedbacks. I'm new and trying to learn Metron as much as possible.

[#6]

> 1. How do I recognize pcap metadata in Elasticsearch indexes (only see yaf, snort, bro, and squid)?

There is not a separate index specifically for pcap metadata. I am just saying that the metadata that you are looking for is likely already provided by an existing sensor like Bro or YAF. For example, what to know who your top talkers are? Any flow-level telemetry, like YAF, will answer that question.

What metadata are you looking for specifically?

[#7]

@nallen the option to reply is not there any more so I'm putting my reply here.

Based on your answer and from reading, my understanding is that YAF and Bro also pick up metadata from PCAP and the data is being indexed under YAF, bro, and snort. I'm trying to use wireshark to read my pcap data, add that pcap file to /opt/pcap-replay, then do a date range query for my pcap data from Metron Dashboard. My only issue is that Metron is not responsive to my query.

I'm interested in these fields in pcap file and i'm hoping they're consider to be part of the metadata

wlan.fc.type_subtype, frame.time, wlan.ra, wlan.da, wlan,ta

As always, appreciate your time and response.

[#8]

I would take a look at aols github page for the project called moloch. It might give you some pointers for indexing into elasticsearch!

[#9]

Thank you @matthew longwell I'll look into https://github.com/aol/moloch

[#10]

Hi, Moloch is not the right answer here. Metron stores PCAP in HDFS. ElasticSearch is not fast enough to keep up with indexing all the packets. What metron does is split your network traffic and you have an option to play this traffic to YAF (netflow), Bro (Layer-7 metadata), and Snort (IDS alerts). So your RAW PCAP is stored in HDFS, while YAF, Bro, and Snort are indexed in ElasticSearch. The idea is that you will pivot from Alerts to Flow data to Layer-7 metadata for your investigation and then once you narrowed down what PCAP traffic you want to look at you would use the Metron PCAP service to load a subset of your PCAP from HDFS into Wireshark if you needed additional forensic information