Hi, I've been struggling with this issue and hoping someone can shed some light here. I'm running Metron 0.7.1 with Bro 2.5.5; Sensors yaf and snort are working and I can see the data in dashboard. Bro is running; sending logs to Kafka topic "bro"; I can see lots of logs when I run the kafka-console-consumer.sh; however, when I check bro parser topology I got nothing. I checked the storm worker log for bro and there were no error and it seems to be idle; 2019-06-03 14:27:56.199 o.a.k.c.p.ProducerConfig Thread-5-parserBolt-executor[8 8] [WARN] The configuration request.required.acks = 1 was supplied but isn't a known config. 2019-06-03 14:27:56.199 o.a.k.c.u.AppInfoParser Thread-5-parserBolt-executor[8 8] [INFO] Kafka version : 0.10.0.2.5.0.0-1245 2019-06-03 14:27:56.199 o.a.k.c.u.AppInfoParser Thread-5-parserBolt-executor[8 8] [INFO] Kafka commitId : dae559f56f07e2cd 2019-06-03 14:27:56.204 o.a.s.d.executor Thread-5-parserBolt-executor[8 8] [INFO] Prepared bolt parserBolt:(8) I used the metron/bin/load_tool.sh to check the throughput of bro and I see numbers; why my bro parser in the storm is not picking up any? Monitoring bro every 10000 ms Summarizing over the last 5 monitoring periods (50000ms) 2019/06/03 15:31:33 - 10 eps throughput measured for bro (Mean: 10, Std Dev: 0) 2019/06/03 15:31:43 - 11 eps throughput measured for bro (Mean: 10, Std Dev: 0) 2019/06/03 15:31:53 - 6 eps throughput measured for bro (Mean: 9, Std Dev: 2) 2019/06/03 15:32:03 - 18 eps throughput measured for bro (Mean: 11, Std Dev: 4) 2019/06/03 15:32:13 - 3 eps throughput measured for bro (Mean: 9, Std Dev: 5) 2019/06/03 15:32:23 - 3 eps throughput measured for bro (Mean: 8, Std Dev: 6) 2019/06/03 15:32:33 - 4 eps throughput measured for bro (Mean: 6, Std Dev: 6) ... View more

Hello all, I'm trying to install Metron 0.6 but it looks like the file structure has changed where we cannot install the sensors based on previous version installation guide. Does anyone know how to install sensors with Metron version 0.6? For example, how to install and configure bro-kafka-plugin? Thank you! ... View more

Hi All, I'm stuck trying to build the latest Metron from the github repo. It looks like it has issue with npm; however my VM, I'm connected to the internet and tried NPM version 3.3 and tried version 6.4 but still getting the same error. Please help! [ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.3:npm (npm ci) on project metron-config: Failed to run task: 'npm ci' failed. (error code 1) -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.3:npm (npm ci) on project metron-config: Failed to run task at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288) at org.apache.maven.cli.MavenCli.main(MavenCli.java:199) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)Caused by: org.apache.maven.plugin.MojoFailureException: Failed to run task at com.github.eirslett.maven.plugins.frontend.mojo.AbstractFrontendMojo.execute(AbstractFrontendMojo.java:95) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)... 20 moreCaused by: com.github.eirslett.maven.plugins.frontend.lib.TaskRunnerException: 'npm ci' failed. (error code 1) at com.github.eirslett.maven.plugins.frontend.lib.NodeTaskExecutor.execute(NodeTaskExecutor.java:60) at com.github.eirslett.maven.plugins.frontend.mojo.NpmMojo.execute(NpmMojo.java:62) at com.github.eirslett.maven.plugins.frontend.mojo.AbstractFrontendMojo.execute(AbstractFrontendMojo.java:89)... 22 more ... View more

Hi all, It seems like we don't have many user or just not enough questions on the topic of Metron. I see that we can deploy a 10 node cluster to AWS with ansible scripts and I've tried it. It works very well. Has anyone try the same approach but with VM cluster on a local network? I would think it's feasible but I'm not very familiar with VMs hypervisor and all. So if anyone has done it or know how, please share your thoughts and approach. Thank you! ... View more

screen-shot-2018-03-23-at-52234-pm.png Hi all, I've been stuck on this problem for over a day and I can't seem to resolve it. I messed around with increasing yarn and mapreduce memory as well as breaking down my timerange for smaller window, but nothing seems to work. I'm trying to export out all my pcap data for two days and stuck on a 1hour window. I even broke it down to 30min and 15mins windows. sudo -su hdfs ./bin/pcap_query.sh fixed -df yyyyMMdd-HHmm -rpf '10000' -st 20180306-1730 -et 20180306-1800 -nr 200 The problem is my mapper tasks keep getting killed by ApplicationMaster. Each of the killed task lasted about 40s seconds and the success ones finished at about 30s second. Please help with any suggestion is appreciated. Container killed by the ApplicationMaster.Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143. 18/03/23 17:19:12 INFO mapreduce.Job: Task Id : attempt_1521838071529_0003_m_000022_2, Status : FAILED Error: java.io.EOFExceptionat java.io.DataInputStream.readFully(DataInputStream.java:197)at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:70)at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:120)at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2439)at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2571)at org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.nextKeyValue(SequenceFileRecordReader.java:72)at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:556)at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:422)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164) 18/03/23 17:19:17 INFO mapreduce.Job:map 100% reduce 100% 18/03/23 17:19:17 INFO mapreduce.Job: Job job_1521838071529_0003 failed with state FAILED due to: Task failed task_1521838071529_0003_m_000020Job failed as tasks failed. failed Maps:1 failed Reduces:0 18/03/23 17:19:17 INFO mapreduce.Job: Counters: 41 File System CountersFILE: Number of bytes read=0 FILE: Number of bytes written=25707046 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations= HDFS: Number of bytes read=4320257853 HDFS: Number of bytes written=0 HDFS: Number of read operations=32 HDFS: Number of large read operations=0 HDFS: Number of write operations=0 Job CountersFailed map tasks=12 Killed map tasks=24 Killed reduce tasks=200 Launched map tasks=26 Other local map tasks=11Data-local map tasks=13 Rack-local map tasks=2Total time spent by all maps in occupied slots (ms)=174434 Total time spent by all reduces in occupied slots (ms)=0Total time spent by all map tasks (ms)=174434Total time spent by all reduce tasks (ms)=0Total vcore-milliseconds taken by all map tasks=174434Total vcore-milliseconds taken by all reduce tasks=0 Total megabyte-milliseconds taken by all map tasks=1428963328 Total megabyte-milliseconds taken by all reduce tasks=0 Map-Reduce FrameworkMap input records=26522800Map output records=119240Map output bytes=24087217Map output materialized bytes=24454354Input split bytes=1240 Combine input records=0 Spilled Records=119240 Failed Shuffles=0 Merged Map outputs=0GC time elapsed (ms)=3410 CPU time spent (ms)=113580Physical memory (bytes) snapshot=21083152384 Virtual memory (bytes) snapshot=72407220224 Total committed heap usage (bytes)=22689087488 File Input Format CountersBytes Read=4320256613org.apache.metron.pcap.mr.PcapJob$PCAP_COUNTERMALFORMED_PACKET_COUNT=61 Exception in thread "main" java.lang.RuntimeException: Unable to complete query due to errors.Please check logs for full errors.at org.apache.metron.pcap.mr.PcapJob.query(PcapJob.java:214)at org.apache.metron.pcap.query.PcapCli.run(PcapCli.java:101)at org.apache.metron.pcap.query.PcapCli.main(PcapCli.java:53)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.apache.hadoop.util.RunJar.run(RunJar.java:233)at org.apache.hadoop.util.RunJar.main(RunJar.java:148) ... View more

Hi, I was curious would it be possible to capture wireless network traffic with Metron? I know Metron uses sensors such as bro, yaf, snort, and pycapa/fastcapa to monitor a network interface for alerts. Does anyone know or have try to capture/monitor Wireless traffic and integrate it into Metron telemetry data flow? Thank you in advance. ... View more

Hi, I notice my pcap parser keeps hanging at the "Turning over and writing" until I restart pcap-parser. Is it something I misconfigured? This is my pcap.properties spout.kafka.topic.pcap=pcap topology.worker.childopts= topology.auto-credentials=[] topology.workers=1 kafka.zk=<hidden> hdfs.sync.every=100 hdfs.replication.factor=1 kafka.security.protocol=PLAINTEXT # One of EARLIEST, LATEST, UNCOMMITTED_EARLIEST, UNCOMMITTED_LATEST kafka.pcap.start=UNCOMMITTED_EARLIEST kafka.pcap.numPackets=1000000000 kafka.pcap.maxTimeMS=0 kafka.pcap.ts_scheme=FROM_KEY kafka.pcap.out=/apps/metron/pcap kafka.pcap.ts_granularity=MICROSECONDSkafka.spout.parallelism=1 Last message from pcap topology log PartitionHDFSWriter [INFO] Turning over and writing to /apps/metron/pcap/pcap_pcap_1519138780354408000_0_pcap-1-1519138859: [duration=1519138780354408000 NS, force=false, initial=true, overDuration=false, tooManyPackets=false] Thank you in advanced. ... View more

Hi, I'm currently using metron 0.4.1 and pycapa. I would like to turn off some of the noises from certain IPs for Pycapa. Is that possible? Thank you! ... View more

Hi, I'm trying to run pcap_query for only 1.5 hours I constantly getting OOME when it's about to write to a pcap file as the result. I tried to modify the memory setting for mapreduce but nothing does the trick so far. mapred.child.java.opts="-Xmx4096" I also tried to reduce a smaller window of exporting but I get no results. screen-shot-2018-02-15-at-24025-pm.png I attached the stack trace below. Any suggestion is appreciated. ... View more

Hi, From the documentation [below] I understand PCAP files that are stored in HDFS sequenceFile has the structure of timestamp as the key and a value of the raw data. However, there are files that ONLY contains keys with no value. If no value is captured; why would we need the key? When would this be useful? A key which is the byte representation of a 64-bit unsigned long representing a time-unit since the unix epoch A value which is the raw packet data without header (either global pcap header or packet header) Thank you in advance for feedbacks. ... View more

Hi, I'm trying to understand these three different timestamp; they don't seem to be in synced 1. pycapa captures network traffic and use the timestamp as the key; the timestamp is in microsecond from epoch 2. the timestamp in the sequence filename is in nanosecond from epoch I try to convert these nanosecond from epoch to human readable time and they are very off. For example taking the time from below...This packet was captured on 2018-02-06 but the timestamp in the filename is way in the future. Is my formula not correct? nanosecond to second is 1000000000. Looking at the code, it looks like PcapCLI scan for the timestamp in the filename when we run our query. Then it runs query against the matching files. [root@hdp-slave-3 0.4.1]# date -d @$((5309030255444767488/1000000000)) Thu Mar 27 22:37:35 EDT 2138 3. the user input of query timestamp (pcap_query -st yyyyMMdd). The timestamp we use to query must match the timestamp in the filename and then in the packets stored in sequenceFile format. -rw-r--r-- 3 storm hdfs575 2018-02-06 13:34 /apps/metron/pcap/pcap_pcap_5309030255444767488_0_pcap-7-1517848461 Please shed some light, I'm confused and seem to make things more complicated than it should be. thank you! ... View more

Hi, I'm using Metron 0.4.2 to run pcap_query.sh. From reading the documents, the default output directory is /tmp if -bop is not specified. I tried using default as well as specifying -bop, but I always get Permission denied. If it's trying to write to /tmp, then any user has permission to write. I also checked hdfs /tmp and that's also a true statement that any user can write. I try to dig into the code, but it's not obvious to me where is it attempting to write the output file. I tried to run as hdfs/metron/storm user, but all ends with the same error. Thank you in advance for your help. 18/02/05 13:37:29 ERROR query.PcapCli: Unable to write filejava.io.FileNotFoundException: pcap-data-201802051835-9b5f3448c9914581a78f705e4c3b4de2+0001.pcap (Permission denied) at java.io.FileOutputStream.open0(Native Method)at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.<init>(FileOutputStream.java:162) at org.apache.metron.pcap.query.ResultsWriter.write(ResultsWriter.java:32) at org.apache.metron.pcap.query.PcapCli.run(PcapCli.java:167) at org.apache.metron.pcap.query.PcapCli.main(PcapCli.java:53) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:233) at org.apache.hadoop.util.RunJar.main(RunJar.java:148) ... View more

Hi all, I'm trying to upgrade metron from 0.4.0 to 0.4.2 by using ambari mpack. Everything starts up fine except for the indexing parser topology and metron rest. I couldn't figure out how to solve this error below. That class exists in the jar included in the script that was executing. So I don't see why it's complaining. Any help is greatly appreciated. File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call raise ExecutionFailed(err_msg, code, out, err) resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/metron/0.4.2/bin/zk_load_configs.sh --zk_quorum hadoop-slave-1:2181,hadoop-slave-2:2181,hadoop-master:2181 --mode PATCH --config_type GLOBAL --patch_file /tmp/metron-global-config-patch.json' returned 1. Error: Could not find or load main class org.apache.metron.common.cli.ConfigurationManager ... View more

Hello all, I've upgraded my ambari successfully to 2.6.1. I'm trying to upgrade HDP; screen-shot-2018-01-24-at-43445-pm.png I see 2.6.4.0 available, but the option to install from the "stack and version" page has been disabled and all the components version are the same as the current version. Then on the page of selecting the versions, I can click on "Install on..." button then I chose my cluster, but it doesn't do anything. screen-shot-2018-01-24-at-43725-pm.png I attached the two screenshots. Any help is greatly appreciated ... View more