Member since
05-26-2020
20
Posts
2
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1045 | 06-05-2020 10:55 AM |
01-21-2021
09:17 AM
Hello, I have files that are in server A. I have the ListSFTP -> FetchSFTP processors setup in server B. My goal is to Fetch the files from server A, so that they are in the pipeline of server B, but after fetching the files, I want to move the files to another remote location, server C. Can anybody help me out with this? I feel like it's a simple solution that I'm missing..
... View more
Labels:
- Labels:
-
Apache NiFi
09-14-2020
10:40 AM
Hi Steven, I tried the method you linked but that didn't work. I also configured the SSLContextService with the keystore and truststore I got from Elasticsearch Cluster. But I get the following error now: Is there anyway to stop hostname verification? Or configure it so hostname verification works correctly? I have blocked out the full hostnames but I can assure you the hostname (atlas201.xx.xxx.xx) is the exact same as the specified CN (atlas.xx.xxx.xx)
... View more
09-11-2020
08:41 AM
Hello, I am trying to configure the StandardSSLContextService controller to work with either the PutElasticsearchHttpRecord or PutElasticsearchRecord processors. The Elasticsearch cluster has been secured by Opendistro for Elasticsearch I am able to do the following below successfully: openssl s_client -connect atlas201.cs:9200 -debug -state -cert esnode.pem -key esnode-key.pem -CAfile root-ca.pem But when I try those keys and certs in the StandardSSLContextService I am unable to get it to work. I am not too familiar with Opendistro as this was already a pre-configured cluster that I am working on. Could someone tell me what the following parameters need to be for me to successfully connect and ingest data into the ES cluster?
... View more
Labels:
- Labels:
-
Apache NiFi
08-04-2020
07:35 AM
@mburgessany suggestions? at the moment I am using SplitRecord -> PutElasticsearchHttpRecord and splitting it into 100,000 but it's pretty slow in comparison
... View more
08-04-2020
07:32 AM
Did you ever find a workaround/solution for this?
... View more
07-28-2020
12:12 PM
I have a processor tree that I'm testing out with large json objects (1GB each). The tree is basically GetFile -> UpdateAttribute (add avro schema name) -> PutElasticsearchHttpRecord But PutElasticsearchHttpRecord gives me the following error (see screenshot) Can anybody tell me what's wrong? I have already assigned enough RAM in bootstrap.conf in NiFi: # JVM memory settings java.arg.2=-Xms16g java.arg.3=-Xmx32g
... View more
Labels:
- Labels:
-
Apache NiFi
06-05-2020
10:55 AM
1 Kudo
Figured it out using GetSFTP
... View more
06-05-2020
06:59 AM
I currently have NiFi setup and running on my master node. I have data coming in to my data node (node2). But I want to setup a NiFi processor to Get or Fetch the data from node2. Which processor do I use and how? Also I have data sorted in folders represented by date as name. So I have data within folders such as 2020-06-03 , 2020-06-04 Is there a way to process the data within these efficiently?
... View more
Labels:
- Labels:
-
Apache NiFi
06-01-2020
06:42 AM
1 Kudo
I would also like to add that for some reason when I did this with CentOS with GNOME desktop, this fails. But if I run a minimal install, this works. I'm not sure if this is because of some python libraries being downloaded for GNOME.. Thank you for getting me to the next steps!
... View more
05-27-2020
01:24 PM
..
[INFO] ------------------------------------------------------------------------
[INFO] Building metron-config 0.7.1
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.0.0:clean (default-clean) @ metron-config ---
[INFO] Deleting /root/metron/metron-interface/metron-config/node (includes = [], excludes = [])
[INFO] Deleting /root/metron/metron-interface/metron-config/node_modules (includes = [], excludes = [])
[INFO]
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-versions) @ metron-config ---
[INFO]
[INFO] --- jacoco-maven-plugin:0.8.3:prepare-agent (default) @ metron-config ---
[INFO] argLine set to -javaagent:/root/.m2/repository/org/jacoco/org.jacoco.agent/0.8.3/org.jacoco.agent-0.8.3-runtime.jar=destfile=/root/metron/metron-interface/metron-config/target/jacoco.exec
[INFO]
[INFO] --- frontend-maven-plugin:1.3:install-node-and-npm (install node and npm) @ metron-config ---
[INFO] Installing node version v9.11.1
[INFO] Unpacking /root/.m2/repository/com/github/eirslett/node/9.11.1/node-9.11.1-linux-x64.tar.gz into /root/metron/metron-interface/metron-config/node/tmp
[INFO] Copying node binary from /root/metron/metron-interface/metron-config/node/tmp/node-v9.11.1-linux-x64/bin/node to /root/metron/metron-interface/metron-config/node/node
[INFO] Installed node locally.
[INFO] Installing npm version 6.2.0
[INFO] Unpacking /root/.m2/repository/com/github/eirslett/npm/6.2.0/npm-6.2.0.tar.gz into /root/metron/metron-interface/metron-config/node/node_modules
[INFO] Installed npm locally.
[INFO]
[INFO] --- frontend-maven-plugin:1.3:npm (npm ci) @ metron-config ---
[INFO] npm not inheriting proxy config from Maven
[INFO] Running 'npm ci' in /root/metron/metron-interface/metron-config
[INFO]
[INFO] > fsevents@1.2.4 install /root/metron/metron-interface/metron-config/node_modules/fsevents
[INFO] > node install
[INFO]
[ERROR] internal/modules/cjs/loader.js:638
[ERROR] throw err;
[ERROR] ^
[ERROR]
[ERROR] Error: Cannot find module '/root/metron/metron-interface/metron-config/node_modules/fsevents/install'
[ERROR] at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
[ERROR] at Function.Module._load (internal/modules/cjs/loader.js:562:25)
[ERROR] at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)
[ERROR] at startup (internal/bootstrap/node.js:283:19)
[ERROR] at bootstrapNodeJSCore (internal/bootstrap/node.js:623:3)
[INFO]
[INFO] > node-sass@4.9.3 install /root/metron/metron-interface/metron-config/node_modules/node-sass
[INFO] > node scripts/install.js
[INFO]
[ERROR] internal/modules/cjs/loader.js:638
[ERROR] throw err;
[ERROR] ^
[ERROR]
[ERROR] Error: Cannot find module '/root/metron/metron-interface/metron-config/node_modules/node-sass/scripts/install.js'
[ERROR] at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
[ERROR] at Function.Module._load (internal/modules/cjs/loader.js:562:25)
[ERROR] at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)
[ERROR] at startup (internal/bootstrap/node.js:283:19)
[ERROR] at bootstrapNodeJSCore (internal/bootstrap/node.js:623:3)
[INFO]
[INFO] > puppeteer@1.8.0 install /root/metron/metron-interface/metron-config/node_modules/puppeteer
[INFO] > node install.js
[INFO]
[ERROR] internal/modules/cjs/loader.js:638
[ERROR] throw err;
[ERROR] ^
[ERROR]
[ERROR] Error: Cannot find module '/root/metron/metron-interface/metron-config/node_modules/puppeteer/install.js'
[ERROR] at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
[ERROR] at Function.Module._load (internal/modules/cjs/loader.js:562:25)
[ERROR] at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)
[ERROR] at startup (internal/bootstrap/node.js:283:19)
[ERROR] at bootstrapNodeJSCore (internal/bootstrap/node.js:623:3)
[ERROR] npm ERR! code ELIFECYCLE
[ERROR] npm ERR! errno 1
[ERROR] npm ERR! puppeteer@1.8.0 install: `node install.js`
[ERROR] npm ERR! Exit status 1
[ERROR] npm ERR!
[ERROR] npm ERR! Failed at the puppeteer@1.8.0 install script.
[ERROR] npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
[ERROR]
[ERROR] npm ERR! A complete log of this run can be found in:
[ERROR] npm ERR! /root/.npm/_logs/2020-05-27T20_14_01_068Z-debug.log
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Metron ............................................. SUCCESS [ 0.867 s]
[INFO] metron-stellar ..................................... SUCCESS [ 0.057 s]
[INFO] stellar-common ..................................... SUCCESS [ 8.449 s]
[INFO] metron-analytics ................................... SUCCESS [ 0.016 s]
[INFO] metron-maas-common ................................. SUCCESS [ 0.461 s]
[INFO] metron-platform .................................... SUCCESS [ 0.015 s]
[INFO] metron-zookeeper ................................... SUCCESS [ 0.121 s]
[INFO] metron-test-utilities .............................. SUCCESS [ 0.590 s]
[INFO] metron-integration-test ............................ SUCCESS [ 6.782 s]
[INFO] metron-maas-service ................................ SUCCESS [ 0.687 s]
[INFO] metron-common ...................................... SUCCESS [ 2.304 s]
[INFO] metron-statistics .................................. SUCCESS [ 0.503 s]
[INFO] metron-common-streaming ............................ SUCCESS [ 0.006 s]
[INFO] metron-common-storm ................................ SUCCESS [ 0.292 s]
[INFO] metron-hbase ....................................... SUCCESS [ 0.475 s]
[INFO] metron-enrichment .................................. SUCCESS [ 0.055 s]
[INFO] metron-enrichment-common ........................... SUCCESS [ 1.591 s]
[INFO] metron-writer ...................................... SUCCESS [ 0.013 s]
[INFO] metron-writer-common ............................... SUCCESS [ 0.705 s]
[INFO] metron-writer-storm ................................ SUCCESS [ 0.668 s]
[INFO] metron-storm-kafka-override ........................ SUCCESS [ 0.150 s]
[INFO] metron-storm-kafka ................................. SUCCESS [ 0.302 s]
[INFO] metron-profiler-common ............................. SUCCESS [ 0.372 s]
[INFO] metron-profiler-client ............................. SUCCESS [ 0.891 s]
[INFO] metron-profiler-storm .............................. SUCCESS [ 0.368 s]
[INFO] metron-profiler-spark .............................. SUCCESS [ 1.248 s]
[INFO] metron-profiler-repl ............................... SUCCESS [ 0.205 s]
[INFO] metron-hbase-client ................................ SUCCESS [ 0.012 s]
[INFO] metron-enrichment-storm ............................ SUCCESS [ 0.465 s]
[INFO] metron-indexing .................................... SUCCESS [ 0.680 s]
[INFO] metron-solr ........................................ SUCCESS [ 1.021 s]
[INFO] metron-job ......................................... SUCCESS [ 0.218 s]
[INFO] metron-pcap ........................................ SUCCESS [ 1.105 s]
[INFO] metron-pcap-backend ................................ SUCCESS [ 0.544 s]
[INFO] metron-data-management ............................. SUCCESS [ 1.130 s]
[INFO] metron-parsing ..................................... SUCCESS [ 0.004 s]
[INFO] metron-parsers-common .............................. SUCCESS [ 0.531 s]
[INFO] metron-parsers ..................................... SUCCESS [ 0.693 s]
[INFO] metron-management .................................. SUCCESS [ 1.067 s]
[INFO] elasticsearch-shaded ............................... SUCCESS [ 0.265 s]
[INFO] metron-elasticsearch ............................... SUCCESS [ 0.796 s]
[INFO] metron-parsing-storm ............................... SUCCESS [ 0.634 s]
[INFO] metron-hbase-server ................................ SUCCESS [ 0.253 s]
[INFO] metron-deployment .................................. SUCCESS [ 0.003 s]
[INFO] Elasticsearch Ambari Management Pack ............... SUCCESS [ 0.038 s]
[INFO] Metron Ambari Management Pack ...................... SUCCESS [ 0.131 s]
[INFO] metron-contrib ..................................... SUCCESS [ 0.005 s]
[INFO] metron-docker ...................................... SUCCESS [ 0.008 s]
[INFO] metron-performance ................................. SUCCESS [ 0.453 s]
[INFO] metron-interface ................................... SUCCESS [ 0.002 s]
[INFO] metron-config ...................................... FAILURE [ 12.594 s]
[INFO] metron-alerts ...................................... SKIPPED
[INFO] metron-rest-client ................................. SKIPPED
[INFO] metron-rest ........................................ SKIPPED
[INFO] site-book .......................................... SKIPPED
[INFO] 3rd party Functions (just for tests) ............... SKIPPED
[INFO] stellar-zeppelin ................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 51.478 s
[INFO] Finished at: 2020-05-27T16:14:01-04:00
[INFO] Final Memory: 156M/1300M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.3:npm (npm ci) on project metron-config: Failed to run task: 'npm ci' failed. (error code 1) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :metron-config Unable to get any of the single-node VMs to work. Updated npm, nodejs as well but still getting the same error..
... View more
Labels:
- Labels:
-
Apache Metron