Re: Ingest Yaf to Metron

adroot16 · ‎07-15-2017

@Rahul P You try to the guide bro plugin. But you change ["metadata.broker.list"] = "localhost:9092" to ["metadata.broker.list"] = "node1:6667"

adroot16 · ‎07-14-2017

Hi @Rahul P you can use bro-plugin-kafka for bro and flume for snort to forward events to metron.

adroot16 · ‎06-29-2017

Hi @asubramanian., If I start yaf by command: /etc/init.d/yaf start And then I running command: nohub /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic yaf & It's show log [2017-06-29 10:19:32,326] ERROR Error when sending message to topic yaf with key: null, value: 52 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.TimeoutException: Batch containing 190 record(s) expired due to timeout while requesting metadata from brokers for yaf-0

adroot16 · ‎06-29-2017

Hi @asubramanian, When I run this command it have output. nohup /usr/local/bin/yaf --in eth0 --become-user yaf --live pcap | /usr/local/bin/yafscii --tabular But when I run this command it don't have output nohup /usr/local/bin/yaf --in eth0 --become-user yaf --live pcap | /usr/local/bin/yafscii --tabular | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic yaf &

adroot16 · ‎06-22-2017

Hi @asubramanian, I try to follow your suggest but it's not ingest yaf.

adroot16 · ‎06-10-2017

I configured Yaf on node instance. And I run command to ingest Yaf event to Kafka. When I check storm log on Metron node but I don't find yaf. nohup /usr/local/bin/yaf --in eth0 --become-user yaf --live pcap | /usr/local/bin/yafscii --tabular | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic yaf & Can you help me?

adroot16 · ‎06-08-2017

Hi @asubramanian I re-configured sucessfull. Thanks you.

adroot16 · ‎06-08-2017

Hi @asubramanian Can you susgest help me?

adroot16 · ‎06-07-2017

I think. I miss configure at parserConfig or miss snort pattern.

adroot16 · ‎06-07-2017

You check help me. please. 2017-06-07 17:09:32.589 o.a.m.p.s.BasicSnortParser [ERROR] Unable to parse message: [**] [1:10000001:1] ICMP test detected [**] java.lang.IllegalArgumentException: Unexpected number of fields, expected: 27 in [**] [1:10000001:1] ICMP test detected [**] at org.apache.metron.parsers.snort.BasicSnortParser.parse(BasicSnortParser.java:148) [stormjar.jar:?] at org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:45) [stormjar.jar:?] at org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:123) [stormjar.jar:?] at org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37] at org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37] at org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37] at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37] at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37] at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37] at org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37] at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37] at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77] 2017-06-07 17:09:32.594 o.a.s.d.executor [ERROR] java.lang.IllegalStateException: Unable to parse message: [**] [1:10000001:1] ICMP test detected [**] at org.apache.metron.parsers.snort.BasicSnortParser.parse(BasicSnortParser.java:180) ~[stormjar.jar:?] at org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:45) ~[stormjar.jar:?] at org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:123) [stormjar.jar:?] at org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37] at org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37] at org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37] at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37] at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37] at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37] at org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37] at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37] at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77] Caused by: java.lang.IllegalArgumentException: Unexpected number of fields, expected: 27 in [**] [1:10000001:1] ICMP test detected [**] at org.apache.metron.parsers.snort.BasicSnortParser.parse(BasicSnortParser.java:148) ~[stormjar.jar:?] ... 12 more

adroot16 · ‎06-07-2017

Hi @asubramanian I re-configured my snort system and It's show alert log. [**] [1:10000001:1] ICMP test detected [**] [Classification: Generic ICMP event] [Priority: 3] 06/07/17-16:37:15.044404 172.16.1.10 -> 172.16.1.20 ICMP TTL:126 TOS:0x0 ID:14129 IpLen:20 DgmLen:60 Type:8 Code:0 ID:1 Seq:1523 ECHO And I re-configured snort.json file { "parserClassName":"org.apache.metron.parsers.snort.BasicSnortParser", "sensorTopic":"snort", "parserConfig": { "dateFormat" : "MM/dd/yy-HH:mm:ss.SSSSSS", "timeZone" : "America/New_York" } } But it still fails.

adroot16 · ‎06-07-2017

I tested Snort alert and it's have log info following [**] [1:10000001:1] ICMP test detected [**] [Classification: Generic ICMP event] [Priority: 3] 06/06-14:54:02.125421 172.16.1.10 -> 172.16.1.20 ICMP TTL:126 TOS:0x0 ID:15052 IpLen:20 DgmLen:60 Type:8 Code:0 ID:1 Seq:1473 ECHO When I checked storm log and it's show 2017-06-07 09:39:41.083 o.a.s.d.executor [ERROR] java.lang.IllegalStateException: Unable to parse message: 06/06-14:54:02.125421 172.16.1.10 -> 172.16.1.20 Can you help me?

adroot16 · ‎06-05-2017

I build YAF on node1 and Metron on node2. How to push YAF event to Metron? Can you suggest for me?

adroot16 · ‎05-31-2017

Hi @nallen Thank you. I install YAF, Snort on other nodes. How to push log data to Metron. Can you help me?

adroot16 · ‎05-25-2017

Hi @Jon Zeolla. Thank you for your answer. I set up separate servers Snort, YAF, Bro. How to I ingest log from that servers into metron?

adroot16 · ‎05-24-2017

Hi everyone. I configured Apache Metron full dev 0.3 but I don't push Snort, Bro, Yaf log to Metron Server. Can you help me?

adroot16 · ‎04-18-2017

When I check storm log, it's show. 2017-04-18 17:44:03.921 o.a.s.d.executor [ERROR] java.lang.IllegalStateException: Unable to parse message: 04/18-17:44:00.620823 [**] [1:999158:0] Sample Metron Message from Snort [**] [Priority: 0]

adroot16 · ‎04-18-2017

Hi everyone., I install apache flume and Snort on Server. But Kibana don't index snort*. I configured flume.conf file. snort.channels=memory-channel snort.channels.memory-channel.capacity=1000 snort.channels.memory-channel.transactionCapacity=100 snort.channels.memory-channel.type=memory snort.sinks=kafka-sink logger-sink snort.sinks.kafka-sink.brokerList=node1:6667 snort.sinks.kafka-sink.channel=memory-channel snort.sinks.kafka-sink.topic=snort snort.sinks.kafka-sink.type=org.apache.flume.sink.kafka.KafkaSink snort.sinks.logger-sink.channel=memory-channel snort.sinks.logger-sink.type=logger snort.sources=exec-source snort.sources.exec-source.channels=memory-channel snort.sources.exec-source.command=tail -F /var/log/snort/alert snort.sources.exec-source.logStdErr=true snort.sources.exec-source.restart=true snort.sources.exec-source.type=exec

adroot16 · ‎04-11-2017

Hi @asubramanian, Command "rwfilter --proto=0- --type=all --pass=stdout | rwcut | tail" showed results. But I can't push data from YAF Server to Metron.

adroot16 · ‎04-08-2017

Hi @asubramanian, Thank you very much. I'm have a problem. Wish you counseling help me. I configured YAF Server and I searched Google but I don't push YAF log to Metron.

adroot16 · ‎04-07-2017

Hi everyone. I deploy a Metron cluster using Ambari following article https://community.hortonworks.com/articles/60805/deploying-a-fresh-metron-cluster-using-ambari-serv.html. I pushed Bro logs to Kafka and It's show on Discover. But when I access http://metron:5000 then Kibana dashboard empty. Can you help me?

adroot16 · ‎04-01-2017

Hi @asubramanian I run command on YAF Server but Kibana not pattern index yaf_index*. Can you help me? nohup /usr/local/bin/yaf --silk --ipfix=tcp --live=pcap --out=node1 --ipfix-port=6667 --in=eth0 --applabel --max-payload=384 &

adroot16 · ‎03-31-2017

Hi @nallen. Thank you for your reply. I try to change "format": "strict_date_optional_time||epoch_millis" and delete index but Kibana don't display it properly.

adroot16 · ‎03-31-2017

Hi @asubramanian. Thank you for your reply. I done configured. Index pattern is bro_index_*

adroot16 · ‎03-30-2017

Hi everyone., I want Kibana to show datetime format standard. I push Bro log via Bro Kafka plugin but It show timestamp format. Can you help me?

adroot16 · ‎03-30-2017

I find on /usr/metron/0.3.0/patterns path but It's have asa, common, fireeye, sourcefire, squid, websphere, yaf files.

adroot16 · ‎03-29-2017

Hi @asubramanian, Thanks you. I done configured follow your guide, but I don't pattern Bro log on Kibana. Can you help me?

adroot16 · ‎03-10-2017

Hello @asubramanian I tried your guide. But I don't understand add services for master & slave nodes. Can you help me? Many thanks.

adroot16 · ‎03-10-2017

Hello @Jon Zeolla. I use CentOS 7. I will try your script on CentOS 6.8. How many node do build on your script?

adroot16 · ‎03-01-2017

Hi @asubramanian Thank you for answering my questions. I try configure follow your guide

Online	Offline
Last Visited	‎07-15-2017 02:26 AM

Date Registered	‎03-01-2017 06:37 AM
Last Visited	‎07-15-2017 02:26 AM
Total Messages Posted	31
Total Kudos Received	1

Re: Ingest Yaf to Metron

Re: Ingest Yaf to Metron

Re: Ingest Yaf to Metron

Re: Ingest Yaf to Metron

Re: Ingest Yaf to Metron

Ingest Yaf to Metron

Re: Snort parser

Re: Snort parser

Re: Snort parser

Re: Snort parser

Re: Snort parser

Snort parser

Re: How to push Snort, Bro, YAF log to Metron

Re: How to push Snort, Bro, YAF log to Metron

Re: How to push Snort, Bro, YAF log to Metron

How to push Snort, Bro, YAF log to Metron

Re: Ingest snort alert

Ingest snort alert

Re: Kibana dashboard empty on Metron 0.3

Re: Kibana dashboard empty on Metron 0.3

Kibana dashboard empty on Metron 0.3

Re: How to deploy Metron on CentOS VM of VMware?

Re: Convert timestamp on Kibana

Re: How to deploy Metron on CentOS VM of VMware?

Convert timestamp on Kibana

Re: How to deploy Metron on CentOS VM of VMware?

Re: How to deploy Metron on CentOS VM of VMware?

Re: How to deploy Metron on CentOS VM of VMware?

Re: How to deploy Metron on CentOS VM of VMware?

Re: How to deploy Metron on CentOS VM of VMware?