Is threat intelligence enrichment possible within NiFi itself? I already have GeoIPLookupRecord processor setup and I was wondering if there was a similar way to do any form of threat intelligence on the same data as well? I am trying not to use Metron since it has not worked for me at all.   Any suggestions?   ... View more

I have NiFi set up to publish logs into Kafka topics. The publishing of the logs to Kafka topics is working fine, but when I create a sensor in Metron Management UI, it is not receiving any of the data that is being published into the kafka topics. I can see the logs as well when I use the kafka-console-consumer.sh script.   When I create the metron sensor, it automatically creates a storm parser and I can see it in the Storm UI page. But I do not see any of the logs in Metron or Storm. Is this a kafkaSpout issue? Possible settings within Metron that need to be changed? What do I need to do here?     (as you can see, there are 3 running sensors, but no throughput for the 3 kafka topics)   Can somebody point me in the direction to a solution for this please? ... View more

I am sending relatively large files to GeoEnrichIPRecord processor that are in JSON format. So I have a JsonTreeReader -> AvroSchemaRegistry -> JsonRecordSetWriter   A file that is about 5GB is taking around 5-10mins to process. Is there anything I can do to increase the processing time?   I am trying to get 11TB of data processed, so I need the processing time down significantly.   I have allocated 16GB to JVM memory in bootstrap.conf: # JVM memory settings java.arg.2=-Xms16g java.arg.3=-Xmx16g   I have also changed the maximum timer driven thread count to: 128 (4x CPU cores)   My server hardware specs are as follows: Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz 32 Cores 192 GB RAM   ... View more

I have a TailFile -> PublishKafka processor in NiFi for squid logs. But when I push the logs to Kafka, they don't appear in Elasticsearch. They do show up in Kafka when I run the ./kafka-console-producer.sh script. Storm also sees the new topic I created in Kafka named "squid" But Elasticsearch/Kibana don't see any of the logs.   Is Metron not receiving the logs from Kafka? When I check the sensor in Metron Management UI, it says the Kafka Topic is "emitting" so I assume that Metron does see the logs coming into Kafka?   I am not sure why I don't see the logs in Kibana or when I curl -xget $ELASTICSEARCH:9200   My Kafka Broker, Elasticsearch, Kibana, Metron, Zookeeper are all on the same physical node. Can someone please assist with this? ... View more

Hello,   I currently have files in paths that look like this: /feed/analytics/2020-06-04 /feed/analytics/2020-06-20 /feed/processing/2020-06-04 /feed/processing/2020-06-15 /feed/graphing/2020-06-04   And so on, and I am trying to use ListSFTP to list the files within these directories and FetchSFTP to fetch the files. I tried the following with ListSFTP but it failed: If someone could point me out to the right regex I would appreciate it. Thanks! ... View more

Is it possible to geo enrich bro/zeek logs that are currently in this format? :   timestamp | uid | orig_ip | orig_p | resp_ip | resp_p | protocol | service | duration   If yes, then what is the easiest way to approach this?   If not, am I supposed to convert each entry into a JSON object, and then use GeoEnrichIP?   I was trying to do use the LookUpRecord processor as per the instructions here: https://community.cloudera.com/t5/Community-Articles/Geo-Enrich-NiFi-Provenance-Event-Data-using-LookupRecord/ta-p/247107   But I don't have a JSON object coming into it, so I don't know how to handle this. Could someone please help me out?   ... View more

Hello, I currently have about 15GB of data that is split into conn, weird, and notice log.gz files. I am trying to figure out how to stream this data into Metron. Currently I have a NiFi Processor setup using ListSFTP, FetchSFTP and then decompressing the files. I am not sure what to do next to get the data enriched through Metron. Am I supposed to create Kibana indexes for the logs? Please help me to proceed further with this.   Thanks ... View more

    [INFO] Building metron-config 0.7.1 [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:3.0.0:clean (default-clean) @ metron-config --- [INFO] [INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-versions) @ metron-config --- [INFO] [INFO] --- jacoco-maven-plugin:0.8.3:prepare-agent (default) @ metron-config --- [INFO] argLine set to -javaagent:/root/.m2/repository/org/jacoco/org.jacoco.agent/0.8.3/org.jacoco.agent-0.8.3-runtime.jar=destfile=/root/metron/metron-interface/metron-config/target/jacoco.exec [INFO] [INFO] --- frontend-maven-plugin:1.3:install-node-and-npm (install node and npm) @ metron-config --- [INFO] Installing node version v9.11.1 [INFO] Downloading https://nodejs.org/dist/v9.11.1/node-v9.11.1-linux-x64.tar.gz to /root/.m2/repository/com/github/eirslett/node/9.11.1/node-9.11.1-linux-x64.tar.gz [INFO] No proxies configured [INFO] No proxy was configured, downloading directly [INFO] Unpacking /root/.m2/repository/com/github/eirslett/node/9.11.1/node-9.11.1-linux-x64.tar.gz into /root/metron/metron-interface/metron-config/node/tmp [INFO] Copying node binary from /root/metron/metron-interface/metron-config/node/tmp/node-v9.11.1-linux-x64/bin/node to /root/metron/metron-interface/metron-config/node/node [INFO] Installed node locally. [INFO] Installing npm version 6.2.0 [INFO] Downloading http://registry.npmjs.org/npm/-/npm-6.2.0.tgz to /root/.m2/repository/com/github/eirslett/npm/6.2.0/npm-6.2.0.tar.gz [INFO] No proxies configured [INFO] No proxy was configured, downloading directly [INFO] Unpacking /root/.m2/repository/com/github/eirslett/npm/6.2.0/npm-6.2.0.tar.gz into /root/metron/metron-interface/metron-config/node/node_modules [INFO] Installed npm locally. [INFO] [INFO] --- frontend-maven-plugin:1.3:npm (npm ci) @ metron-config --- [INFO] npm not inheriting proxy config from Maven [INFO] Running 'npm ci' in /root/metron/metron-interface/metron-config [INFO] [INFO] > fsevents@1.2.4 install /root/metron/metron-interface/metron-config/node_modules/fsevents [INFO] > node install [INFO] [ERROR] internal/modules/cjs/loader.js:638 [ERROR] throw err; [ERROR] ^ [ERROR] [ERROR] Error: Cannot find module '/root/metron/metron-interface/metron-config/node_modules/fsevents/install' [ERROR] at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15) [ERROR] at Function.Module._load (internal/modules/cjs/loader.js:562:25) [ERROR] at Function.Module.runMain (internal/modules/cjs/loader.js:831:12) [ERROR] at startup (internal/bootstrap/node.js:283:19) [ERROR] at bootstrapNodeJSCore (internal/bootstrap/node.js:623:3) [INFO] [INFO] > node-sass@4.9.3 install /root/metron/metron-interface/metron-config/node_modules/node-sass [INFO] > node scripts/install.js [INFO] [ERROR] internal/modules/cjs/loader.js:638 [ERROR] throw err; [ERROR] ^ [ERROR] [ERROR] Error: Cannot find module '/root/metron/metron-interface/metron-config/node_modules/node-sass/scripts/install.js' [ERROR] at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15) [ERROR] at Function.Module._load (internal/modules/cjs/loader.js:562:25) [ERROR] at Function.Module.runMain (internal/modules/cjs/loader.js:831:12) [ERROR] at startup (internal/bootstrap/node.js:283:19) [ERROR] at bootstrapNodeJSCore (internal/bootstrap/node.js:623:3) [INFO] [INFO] > puppeteer@1.8.0 install /root/metron/metron-interface/metron-config/node_modules/puppeteer [INFO] > node install.js [INFO] [ERROR] internal/modules/cjs/loader.js:638 [ERROR] throw err; [ERROR] ^ [ERROR] [ERROR] Error: Cannot find module '/root/metron/metron-interface/metron-config/node_modules/puppeteer/install.js' [ERROR] at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15) [ERROR] at Function.Module._load (internal/modules/cjs/loader.js:562:25) [ERROR] at Function.Module.runMain (internal/modules/cjs/loader.js:831:12) [ERROR] at startup (internal/bootstrap/node.js:283:19) [ERROR] at bootstrapNodeJSCore (internal/bootstrap/node.js:623:3) [ERROR] npm ERR! code ELIFECYCLE [ERROR] npm ERR! errno 1 [ERROR] npm ERR! puppeteer@1.8.0 install: `node install.js` [ERROR] npm ERR! Exit status 1 [ERROR] npm ERR! [ERROR] npm ERR! Failed at the puppeteer@1.8.0 install script. [ERROR] npm ERR! This is probably not a problem with npm. There is likely additional logging output above. [ERROR] [ERROR] npm ERR! A complete log of this run can be found in: [ERROR] npm ERR! /root/.npm/_logs/2020-05-27T19_33_40_941Z-debug.log [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Metron ............................................. SUCCESS [ 8.174 s] [INFO] metron-stellar ..................................... SUCCESS [ 0.561 s] [INFO] stellar-common ..................................... SUCCESS [ 43.744 s] [INFO] metron-analytics ................................... SUCCESS [ 0.140 s] [INFO] metron-maas-common ................................. SUCCESS [ 0.707 s] [INFO] metron-platform .................................... SUCCESS [ 3.832 s] [INFO] metron-zookeeper ................................... SUCCESS [ 0.298 s] [INFO] metron-test-utilities .............................. SUCCESS [ 0.489 s] [INFO] metron-integration-test ............................ SUCCESS [ 6.854 s] [INFO] metron-maas-service ................................ SUCCESS [ 4.830 s] [INFO] metron-common ...................................... SUCCESS [ 18.655 s] [INFO] metron-statistics .................................. SUCCESS [ 15.485 s] [INFO] metron-common-streaming ............................ SUCCESS [ 0.095 s] [INFO] metron-common-storm ................................ SUCCESS [ 0.896 s] [INFO] metron-hbase ....................................... SUCCESS [ 5.999 s] [INFO] metron-enrichment .................................. SUCCESS [ 0.178 s] [INFO] metron-enrichment-common ........................... SUCCESS [ 28.021 s] [INFO] metron-writer ...................................... SUCCESS [ 0.102 s] [INFO] metron-writer-common ............................... SUCCESS [ 14.084 s] [INFO] metron-writer-storm ................................ SUCCESS [ 0.922 s] [INFO] metron-storm-kafka-override ........................ SUCCESS [ 6.694 s] [INFO] metron-storm-kafka ................................. SUCCESS [ 0.457 s] [INFO] metron-profiler-common ............................. SUCCESS [ 0.913 s] [INFO] metron-profiler-client ............................. SUCCESS [ 19.419 s] [INFO] metron-profiler-storm .............................. SUCCESS [ 44.677 s] [INFO] metron-profiler-spark .............................. SUCCESS [02:20 min] [INFO] metron-profiler-repl ............................... SUCCESS [ 13.246 s] [INFO] metron-hbase-client ................................ SUCCESS [ 6.349 s] [INFO] metron-enrichment-storm ............................ SUCCESS [ 49.006 s] [INFO] metron-indexing .................................... SUCCESS [ 21.283 s] [INFO] metron-solr ........................................ SUCCESS [01:27 min] [INFO] metron-job ......................................... SUCCESS [ 0.424 s] [INFO] metron-pcap ........................................ SUCCESS [ 1.544 s] [INFO] metron-pcap-backend ................................ SUCCESS [ 25.809 s] [INFO] metron-data-management ............................. SUCCESS [ 26.077 s] [INFO] metron-parsing ..................................... SUCCESS [ 0.077 s] [INFO] metron-parsers-common .............................. SUCCESS [ 49.663 s] [INFO] metron-parsers ..................................... SUCCESS [ 44.046 s] [INFO] metron-management .................................. SUCCESS [ 7.935 s] [INFO] elasticsearch-shaded ............................... SUCCESS [ 36.180 s] [INFO] metron-elasticsearch ............................... SUCCESS [ 46.452 s] [INFO] metron-parsing-storm ............................... SUCCESS [ 46.663 s] [INFO] metron-hbase-server ................................ SUCCESS [ 10.589 s] [INFO] metron-deployment .................................. SUCCESS [ 0.087 s] [INFO] Elasticsearch Ambari Management Pack ............... SUCCESS [ 0.107 s] [INFO] Metron Ambari Management Pack ...................... SUCCESS [ 0.373 s] [INFO] metron-contrib ..................................... SUCCESS [ 0.065 s] [INFO] metron-docker ...................................... SUCCESS [ 0.486 s] [INFO] metron-performance ................................. SUCCESS [ 2.768 s] [INFO] metron-interface ................................... SUCCESS [ 0.085 s] [INFO] metron-config ...................................... FAILURE [ 40.080 s] [INFO] metron-alerts ...................................... SKIPPED [INFO] metron-rest-client ................................. SKIPPED [INFO] metron-rest ........................................ SKIPPED [INFO] site-book .......................................... SKIPPED [INFO] 3rd party Functions (just for tests) ............... SKIPPED [INFO] stellar-zeppelin ................................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 14:50 min [INFO] Finished at: 2020-05-27T15:33:40-04:00 [INFO] Final Memory: 218M/2196M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.3:npm (npm ci) on project metron-config: Failed to run task: 'npm ci' failed. (error code 1) -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :metron-config       Here is the output of platform-info.sh script   Metron 0.7.1 fatal: Not a git repository (or any of the parent directories): .git -- ansible 2.9.9 config file = /etc/ansible/ansible.cfg configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python2.7/site-packages/ansible executable location = /bin/ansible python version = 2.7.5 (default, Apr 2 2020, 13:16:51) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] -- Vagrant 2.2.9 -- vagrant-hostmanager (1.8.9, global) -- Python 2.7.5 -- Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T11:41:47-05:00) Maven home: /opt/apache-maven-3.3.9 Java version: 1.8.0_252, vendor: Oracle Corporation Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-1127.8.2.el7.x86_64", arch: "amd64", family: "unix" -- Docker version 1.13.1, build 64e9980/1.13.1 -- node v10.20.1 -- npm 6.14.4 -- g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39) Copyright (C) 2015 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. -- Compiler is C++11 compliant -- Linux scslt227 3.10.0-1127.8.2.el7.x86_64 #1 SMP Tue May 12 16:57:42 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux -- Total System Memory = 19681.2 MB Processor Model: Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz Processor Speed: 3099.957 MHz Total Physical Processors: 4 Total cores: 8 Disk information: /dev/mapper/centos-root 66G 15G 52G 22% / /dev/nvme0n1p5 1014M 227M 788M 23% /boot /dev/nvme0n1p1 200M 12M 189M 6% /boot/efi /dev/mapper/centos-home 21G 360M 21G 2% /home This CPU appears to support virtualization     I have also tried the single-node VM using vagrant and it also stops at metron-config. Could anyone help with this? I've been trying for quite a few days now scouring through the forums.. ... View more