Hello,   I have files that are in server A. I have the ListSFTP -> FetchSFTP processors setup in server B. My goal is to Fetch the files from server A, so that they are in the pipeline of server B, but after fetching the files, I want to move the files to another remote location, server C.   Can anybody help me out with this? I feel like it's a simple solution that I'm missing..     ... View more

Hello, I am trying to configure the StandardSSLContextService controller to work with either the PutElasticsearchHttpRecord or PutElasticsearchRecord processors. The Elasticsearch cluster has been secured by Opendistro for Elasticsearch I am able to do the following below successfully: openssl s_client -connect atlas201.cs:9200 -debug -state -cert esnode.pem -key esnode-key.pem -CAfile root-ca.pem   But when I try those keys and certs in the StandardSSLContextService I am unable to get it to work. I am not too familiar with Opendistro as this was already a pre-configured cluster that I am working on. Could someone tell me what the following parameters need to be for me to successfully connect and ingest data into the ES cluster?   ... View more

I have a processor tree that I'm testing out with large json objects (1GB each). The tree is basically GetFile -> UpdateAttribute (add avro schema name) -> PutElasticsearchHttpRecord   But PutElasticsearchHttpRecord gives me the following error (see screenshot)     Can anybody tell me what's wrong? I have already assigned enough RAM in bootstrap.conf in NiFi: # JVM memory settings java.arg.2=-Xms16g java.arg.3=-Xmx32g       ... View more

Is threat intelligence enrichment possible within NiFi itself? I already have GeoIPLookupRecord processor setup and I was wondering if there was a similar way to do any form of threat intelligence on the same data as well? I am trying not to use Metron since it has not worked for me at all.   Any suggestions?   ... View more

I have NiFi set up to publish logs into Kafka topics. The publishing of the logs to Kafka topics is working fine, but when I create a sensor in Metron Management UI, it is not receiving any of the data that is being published into the kafka topics. I can see the logs as well when I use the kafka-console-consumer.sh script.   When I create the metron sensor, it automatically creates a storm parser and I can see it in the Storm UI page. But I do not see any of the logs in Metron or Storm. Is this a kafkaSpout issue? Possible settings within Metron that need to be changed? What do I need to do here?     (as you can see, there are 3 running sensors, but no throughput for the 3 kafka topics)   Can somebody point me in the direction to a solution for this please? ... View more

I am sending relatively large files to GeoEnrichIPRecord processor that are in JSON format. So I have a JsonTreeReader -> AvroSchemaRegistry -> JsonRecordSetWriter   A file that is about 5GB is taking around 5-10mins to process. Is there anything I can do to increase the processing time?   I am trying to get 11TB of data processed, so I need the processing time down significantly.   I have allocated 16GB to JVM memory in bootstrap.conf: # JVM memory settings java.arg.2=-Xms16g java.arg.3=-Xmx16g   I have also changed the maximum timer driven thread count to: 128 (4x CPU cores)   My server hardware specs are as follows: Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz 32 Cores 192 GB RAM   ... View more

I have a TailFile -> PublishKafka processor in NiFi for squid logs. But when I push the logs to Kafka, they don't appear in Elasticsearch. They do show up in Kafka when I run the ./kafka-console-producer.sh script. Storm also sees the new topic I created in Kafka named "squid" But Elasticsearch/Kibana don't see any of the logs.   Is Metron not receiving the logs from Kafka? When I check the sensor in Metron Management UI, it says the Kafka Topic is "emitting" so I assume that Metron does see the logs coming into Kafka?   I am not sure why I don't see the logs in Kibana or when I curl -xget $ELASTICSEARCH:9200   My Kafka Broker, Elasticsearch, Kibana, Metron, Zookeeper are all on the same physical node. Can someone please assist with this? ... View more

Hello,   I currently have files in paths that look like this: /feed/analytics/2020-06-04 /feed/analytics/2020-06-20 /feed/processing/2020-06-04 /feed/processing/2020-06-15 /feed/graphing/2020-06-04   And so on, and I am trying to use ListSFTP to list the files within these directories and FetchSFTP to fetch the files. I tried the following with ListSFTP but it failed: If someone could point me out to the right regex I would appreciate it. Thanks! ... View more

Is it possible to geo enrich bro/zeek logs that are currently in this format? :   timestamp | uid | orig_ip | orig_p | resp_ip | resp_p | protocol | service | duration   If yes, then what is the easiest way to approach this?   If not, am I supposed to convert each entry into a JSON object, and then use GeoEnrichIP?   I was trying to do use the LookUpRecord processor as per the instructions here: https://community.cloudera.com/t5/Community-Articles/Geo-Enrich-NiFi-Provenance-Event-Data-using-LookupRecord/ta-p/247107   But I don't have a JSON object coming into it, so I don't know how to handle this. Could someone please help me out?   ... View more

Hello, I currently have about 15GB of data that is split into conn, weird, and notice log.gz files. I am trying to figure out how to stream this data into Metron. Currently I have a NiFi Processor setup using ListSFTP, FetchSFTP and then decompressing the files. I am not sure what to do next to get the data enriched through Metron. Am I supposed to create Kibana indexes for the logs? Please help me to proceed further with this.   Thanks ... View more

    [INFO] Building metron-config 0.7.1 [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:3.0.0:clean (default-clean) @ metron-config --- [INFO] [INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-versions) @ metron-config --- [INFO] [INFO] --- jacoco-maven-plugin:0.8.3:prepare-agent (default) @ metron-config --- [INFO] argLine set to -javaagent:/root/.m2/repository/org/jacoco/org.jacoco.agent/0.8.3/org.jacoco.agent-0.8.3-runtime.jar=destfile=/root/metron/metron-interface/metron-config/target/jacoco.exec [INFO] [INFO] --- frontend-maven-plugin:1.3:install-node-and-npm (install node and npm) @ metron-config --- [INFO] Installing node version v9.11.1 [INFO] Downloading https://nodejs.org/dist/v9.11.1/node-v9.11.1-linux-x64.tar.gz to /root/.m2/repository/com/github/eirslett/node/9.11.1/node-9.11.1-linux-x64.tar.gz [INFO] No proxies configured [INFO] No proxy was configured, downloading directly [INFO] Unpacking /root/.m2/repository/com/github/eirslett/node/9.11.1/node-9.11.1-linux-x64.tar.gz into /root/metron/metron-interface/metron-config/node/tmp [INFO] Copying node binary from /root/metron/metron-interface/metron-config/node/tmp/node-v9.11.1-linux-x64/bin/node to /root/metron/metron-interface/metron-config/node/node [INFO] Installed node locally. [INFO] Installing npm version 6.2.0 [INFO] Downloading http://registry.npmjs.org/npm/-/npm-6.2.0.tgz to /root/.m2/repository/com/github/eirslett/npm/6.2.0/npm-6.2.0.tar.gz [INFO] No proxies configured [INFO] No proxy was configured, downloading directly [INFO] Unpacking /root/.m2/repository/com/github/eirslett/npm/6.2.0/npm-6.2.0.tar.gz into /root/metron/metron-interface/metron-config/node/node_modules [INFO] Installed npm locally. [INFO] [INFO] --- frontend-maven-plugin:1.3:npm (npm ci) @ metron-config --- [INFO] npm not inheriting proxy config from Maven [INFO] Running 'npm ci' in /root/metron/metron-interface/metron-config [INFO] [INFO] > fsevents@1.2.4 install /root/metron/metron-interface/metron-config/node_modules/fsevents [INFO] > node install [INFO] [ERROR] internal/modules/cjs/loader.js:638 [ERROR] throw err; [ERROR] ^ [ERROR] [ERROR] Error: Cannot find module '/root/metron/metron-interface/metron-config/node_modules/fsevents/install' [ERROR] at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15) [ERROR] at Function.Module._load (internal/modules/cjs/loader.js:562:25) [ERROR] at Function.Module.runMain (internal/modules/cjs/loader.js:831:12) [ERROR] at startup (internal/bootstrap/node.js:283:19) [ERROR] at bootstrapNodeJSCore (internal/bootstrap/node.js:623:3) [INFO] [INFO] > node-sass@4.9.3 install /root/metron/metron-interface/metron-config/node_modules/node-sass [INFO] > node scripts/install.js [INFO] [ERROR] internal/modules/cjs/loader.js:638 [ERROR] throw err; [ERROR] ^ [ERROR] [ERROR] Error: Cannot find module '/root/metron/metron-interface/metron-config/node_modules/node-sass/scripts/install.js' [ERROR] at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15) [ERROR] at Function.Module._load (internal/modules/cjs/loader.js:562:25) [ERROR] at Function.Module.runMain (internal/modules/cjs/loader.js:831:12) [ERROR] at startup (internal/bootstrap/node.js:283:19) [ERROR] at bootstrapNodeJSCore (internal/bootstrap/node.js:623:3) [INFO] [INFO] > puppeteer@1.8.0 install /root/metron/metron-interface/metron-config/node_modules/puppeteer [INFO] > node install.js [INFO] [ERROR] internal/modules/cjs/loader.js:638 [ERROR] throw err; [ERROR] ^ [ERROR] [ERROR] Error: Cannot find module '/root/metron/metron-interface/metron-config/node_modules/puppeteer/install.js' [ERROR] at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15) [ERROR] at Function.Module._load (internal/modules/cjs/loader.js:562:25) [ERROR] at Function.Module.runMain (internal/modules/cjs/loader.js:831:12) [ERROR] at startup (internal/bootstrap/node.js:283:19) [ERROR] at bootstrapNodeJSCore (internal/bootstrap/node.js:623:3) [ERROR] npm ERR! code ELIFECYCLE [ERROR] npm ERR! errno 1 [ERROR] npm ERR! puppeteer@1.8.0 install: `node install.js` [ERROR] npm ERR! Exit status 1 [ERROR] npm ERR! [ERROR] npm ERR! Failed at the puppeteer@1.8.0 install script. [ERROR] npm ERR! This is probably not a problem with npm. There is likely additional logging output above. [ERROR] [ERROR] npm ERR! A complete log of this run can be found in: [ERROR] npm ERR! /root/.npm/_logs/2020-05-27T19_33_40_941Z-debug.log [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Metron ............................................. SUCCESS [ 8.174 s] [INFO] metron-stellar ..................................... SUCCESS [ 0.561 s] [INFO] stellar-common ..................................... SUCCESS [ 43.744 s] [INFO] metron-analytics ................................... SUCCESS [ 0.140 s] [INFO] metron-maas-common ................................. SUCCESS [ 0.707 s] [INFO] metron-platform .................................... SUCCESS [ 3.832 s] [INFO] metron-zookeeper ................................... SUCCESS [ 0.298 s] [INFO] metron-test-utilities .............................. SUCCESS [ 0.489 s] [INFO] metron-integration-test ............................ SUCCESS [ 6.854 s] [INFO] metron-maas-service ................................ SUCCESS [ 4.830 s] [INFO] metron-common ...................................... SUCCESS [ 18.655 s] [INFO] metron-statistics .................................. SUCCESS [ 15.485 s] [INFO] metron-common-streaming ............................ SUCCESS [ 0.095 s] [INFO] metron-common-storm ................................ SUCCESS [ 0.896 s] [INFO] metron-hbase ....................................... SUCCESS [ 5.999 s] [INFO] metron-enrichment .................................. SUCCESS [ 0.178 s] [INFO] metron-enrichment-common ........................... SUCCESS [ 28.021 s] [INFO] metron-writer ...................................... SUCCESS [ 0.102 s] [INFO] metron-writer-common ............................... SUCCESS [ 14.084 s] [INFO] metron-writer-storm ................................ SUCCESS [ 0.922 s] [INFO] metron-storm-kafka-override ........................ SUCCESS [ 6.694 s] [INFO] metron-storm-kafka ................................. SUCCESS [ 0.457 s] [INFO] metron-profiler-common ............................. SUCCESS [ 0.913 s] [INFO] metron-profiler-client ............................. SUCCESS [ 19.419 s] [INFO] metron-profiler-storm .............................. SUCCESS [ 44.677 s] [INFO] metron-profiler-spark .............................. SUCCESS [02:20 min] [INFO] metron-profiler-repl ............................... SUCCESS [ 13.246 s] [INFO] metron-hbase-client ................................ SUCCESS [ 6.349 s] [INFO] metron-enrichment-storm ............................ SUCCESS [ 49.006 s] [INFO] metron-indexing .................................... SUCCESS [ 21.283 s] [INFO] metron-solr ........................................ SUCCESS [01:27 min] [INFO] metron-job ......................................... SUCCESS [ 0.424 s] [INFO] metron-pcap ........................................ SUCCESS [ 1.544 s] [INFO] metron-pcap-backend ................................ SUCCESS [ 25.809 s] [INFO] metron-data-management ............................. SUCCESS [ 26.077 s] [INFO] metron-parsing ..................................... SUCCESS [ 0.077 s] [INFO] metron-parsers-common .............................. SUCCESS [ 49.663 s] [INFO] metron-parsers ..................................... SUCCESS [ 44.046 s] [INFO] metron-management .................................. SUCCESS [ 7.935 s] [INFO] elasticsearch-shaded ............................... SUCCESS [ 36.180 s] [INFO] metron-elasticsearch ............................... SUCCESS [ 46.452 s] [INFO] metron-parsing-storm ............................... SUCCESS [ 46.663 s] [INFO] metron-hbase-server ................................ SUCCESS [ 10.589 s] [INFO] metron-deployment .................................. SUCCESS [ 0.087 s] [INFO] Elasticsearch Ambari Management Pack ............... SUCCESS [ 0.107 s] [INFO] Metron Ambari Management Pack ...................... SUCCESS [ 0.373 s] [INFO] metron-contrib ..................................... SUCCESS [ 0.065 s] [INFO] metron-docker ...................................... SUCCESS [ 0.486 s] [INFO] metron-performance ................................. SUCCESS [ 2.768 s] [INFO] metron-interface ................................... SUCCESS [ 0.085 s] [INFO] metron-config ...................................... FAILURE [ 40.080 s] [INFO] metron-alerts ...................................... SKIPPED [INFO] metron-rest-client ................................. SKIPPED [INFO] metron-rest ........................................ SKIPPED [INFO] site-book .......................................... SKIPPED [INFO] 3rd party Functions (just for tests) ............... SKIPPED [INFO] stellar-zeppelin ................................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 14:50 min [INFO] Finished at: 2020-05-27T15:33:40-04:00 [INFO] Final Memory: 218M/2196M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.3:npm (npm ci) on project metron-config: Failed to run task: 'npm ci' failed. (error code 1) -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :metron-config       Here is the output of platform-info.sh script   Metron 0.7.1 fatal: Not a git repository (or any of the parent directories): .git -- ansible 2.9.9 config file = /etc/ansible/ansible.cfg configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python2.7/site-packages/ansible executable location = /bin/ansible python version = 2.7.5 (default, Apr 2 2020, 13:16:51) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] -- Vagrant 2.2.9 -- vagrant-hostmanager (1.8.9, global) -- Python 2.7.5 -- Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T11:41:47-05:00) Maven home: /opt/apache-maven-3.3.9 Java version: 1.8.0_252, vendor: Oracle Corporation Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-1127.8.2.el7.x86_64", arch: "amd64", family: "unix" -- Docker version 1.13.1, build 64e9980/1.13.1 -- node v10.20.1 -- npm 6.14.4 -- g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39) Copyright (C) 2015 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. -- Compiler is C++11 compliant -- Linux scslt227 3.10.0-1127.8.2.el7.x86_64 #1 SMP Tue May 12 16:57:42 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux -- Total System Memory = 19681.2 MB Processor Model: Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz Processor Speed: 3099.957 MHz Total Physical Processors: 4 Total cores: 8 Disk information: /dev/mapper/centos-root 66G 15G 52G 22% / /dev/nvme0n1p5 1014M 227M 788M 23% /boot /dev/nvme0n1p1 200M 12M 189M 6% /boot/efi /dev/mapper/centos-home 21G 360M 21G 2% /home This CPU appears to support virtualization     I have also tried the single-node VM using vagrant and it also stops at metron-config. Could anyone help with this? I've been trying for quite a few days now scouring through the forums.. ... View more