About nnat25191

nnat25191 · ‎11-15-2017

Thank you @jsirota for the explaination. I think I got the first part comfortably. However, the second part is still fuzzy to me where we narrow down to certain data to export out to PCAP format in order to view them in wireshark. I was looking up for Metron meetup around NOVA/MD area, but couldn't find any. There are so much with Metron I would like to learn and understand better. I started to tap into our company network interface instead of the tap0 switch we created and I started to run into more issues with services being down.

nnat25191 · ‎11-01-2017

@jsirota Thank you for your reply and thank you for the information on the next release as well as available tool. I found a workaround to delete in indexes through ElasticSearch plugin via json delete query. Thank you!

nnat25191 · ‎10-30-2017

Thank you @cstella and @jsirota The pcap data stored in HDFS is sequence files. How do you view them in Wireshark? My guess would be somehow get the pcap_inspector service to spit out the result of the filter in PCAP format? Thank you.

nnat25191 · ‎10-17-2017

@Lee Adrian @nallen I also ran into the same issue. I changed ElasticSearch templates as well as storm parser for each data type, but it's still epoch_millis. I wonder do I need to change the sensor configuration itself? The epoch time is not useful to the users b/c they have to convert it

nnat25191 · ‎10-16-2017

Thank you @matthew longwell I'll look into https://github.com/aol/moloch

nnat25191 · ‎10-16-2017

Thank you all feedbacks. Simon, Thank you for pointing out many useful things. It would be helpful if on Metron Document, we makr what's deprecated and what's no longer supported. I would like to understand why Metron doesn't support bringing in PCAP data to add to Metron Cluster. I thought the whole idea is to have the data in one place. My work around would be to use tshark to extract PCAP metadata and push it to Metron manually. What's your take on that? I'll spend more time with query and inspector tools for PCAP, however, we like to visualize our data on a dashboard along with our other network traffic data collected. I do use Ambari to manage services, however it doesn't have status on the sensors like yaf, bro, and snort like Monit does. Unless, I'm missing a configuration/installation steps to add sensor services to Ambari. Thank you for your time and feedbacks.

nnat25191 · ‎10-11-2017

Hi all, I have a metron cluster of 3 nodes running. I also have pycapa, pcap-service, and pcap-replay running. I'm trying to understand the role of these three and it's a bit fuzzy to me. I'm using Metron 0.4 version. Pycapa acts as the Metron probe and capture the network traffic then send them to Kafka. PCAP storm topology picks it up and store them in HDFS as sequence files. What's next? How is this data become available in Metron? who, how and when this data get used? pcap-replay - We put pcap files that were captured previously to for example /opt/pcap-replay/*.pcap, pcap-replay will replay these packets so that Snort, Yaf, and Bro to pick up the data and send them to STORM topology? pcap-service - was this designed only work for OpenSoc and no longer support in Metron 0.4? Thank you for any feedbacks in advance.

nnat25191 · ‎10-10-2017

@nallen the option to reply is not there any more so I'm putting my reply here. Based on your answer and from reading, my understanding is that YAF and Bro also pick up metadata from PCAP and the data is being indexed under YAF, bro, and snort. I'm trying to use wireshark to read my pcap data, add that pcap file to /opt/pcap-replay, then do a date range query for my pcap data from Metron Dashboard. My only issue is that Metron is not responsive to my query. I'm interested in these fields in pcap file and i'm hoping they're consider to be part of the metadata wlan.fc.type_subtype, frame.time, wlan.ra, wlan.da, wlan,ta As always, appreciate your time and response.

nnat25191 · ‎10-06-2017

Hi, I have Metron cluster setup and it's been running. The data in ElasticSearch is starting to fill up pretty quickly and I wanted to use the functionality that came with Metron to prune ElasticSearch index. However, I could not find documentation regarding prune_elasticsearch_indeces.sh and i'm getting exception of ClassNotFound. If you have used this tool before or know how to run it properly or if I'm missing any installation for the missing library, please shed some lights. Thank you! /usr/metron/0.4.0/bin/prune_elasticsearch_indices.sh -s 09/28/17 -n 2 -p bro_ -z master:2181,node-1:2181,node-2:2181 17/10/06 14:55:03 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-master:2181,hadoop-slave-1:2181,hadoop-slave-2:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@6a801d4b 17/10/06 14:55:03 INFO zookeeper.ClientCnxn: Opening socket connection to server hadoop-master/172.16.8.236:2181. Will not attempt to authenticate using SASL (unknown error) 17/10/06 14:55:03 INFO zookeeper.ClientCnxn: Socket connection established to hadoop-master/172.16.8.236:2181, initiating session 17/10/06 14:55:03 INFO zookeeper.ClientCnxn: Session establishment complete on server hadoop-master/172.16.8.236:2181, sessionid = 0x15ef200cb570461, negotiated timeout = 60000 17/10/06 14:55:03 INFO state.ConnectionStateManager: State change: CONNECTED 17/10/06 14:55:04 INFO elasticsearch.plugins: [Bloodstorm] modules [], plugins [], sites [] Exception in thread "main" java.lang.NoSuchMethodError: org.apache.metron.guava.dataload.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor; at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:190) at org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:131) at org.apache.metron.dataloads.bulk.ElasticsearchDataPrunerRunner.main(ElasticsearchDataPrunerRunner.java:103) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:233) at org.apache.hadoop.util.RunJar.main(RunJar.java:148)

nnat25191 · ‎10-04-2017

Wonderful! 1. extending to the same question, there are pcap files in my HDFS, does this mean that the metadata fields are highly possibly made it into ElasticSearch and available? Do I need to create that dataflow/index or is it already in ElasticSearch by kafkaConsumer setup for pcap? How do I recognize pcap metadata in Elasticsearch indexes (only see yaf, snort, bro, and squid)? 2. there seems to be no pcap parser available in metron sensor rest ui. There are only parsers for bro, snort, yaf, websphere, asa, JsonMap, and Grok. What do you recommend for Pcap parser? I used JsonMap and I don't see error in the log, but I don't know if it parse correctly for metadata fields. Again, thank you for your feedbacks. I'm new and trying to learn Metron as much as possible.

Online	Offline
Last Visited	‎06-13-2019 03:53 PM

Member Since	‎01-08-2017 10:33 PM
Last Visited	‎06-13-2019 03:53 PM
Posts	79
Kudos received	6

Cloudera Community

Re: Metron 4.2.0 deployment to AWS

Re: Pcap data index to ElasticSearch

Re: metron script prune_elasticsearch_indeces.sh

Re: Metron pcap analysis vs wireshark

Re: Convert timestamp on Kibana

Re: Pcap data index to ElasticSearch

Re: metron pcap data stored in HDFS sequence forma...

metron pcap data stored in HDFS sequence format

Re: Pcap data index to ElasticSearch

metron script prune_elasticsearch_indeces.sh

Re: Pcap data index to ElasticSearch