Member since
10-27-2016
122
Posts
13
Kudos Received
3
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
836 | 01-08-2017 07:14 PM | |
2586 | 12-07-2016 06:17 PM | |
2658 | 11-08-2016 08:49 PM |
08-06-2019
03:09 PM
Hi, I have Minifi on a windows machine. However, I am unable to use the config.bat transform command. I get the error: Error: Could not find or load main class Files\minifi-0.5.0-bin\minifi-toolkit-0.5.0\bin\..\lib\* Im not sure what it means. Thank you
... View more
Labels:
- Labels:
-
Apache MiNiFi
-
Apache NiFi
01-31-2018
05:07 PM
Hi, yea its a bit tricky. Thanks for your time. Much appreciated.
... View more
01-30-2018
03:18 PM
1 Kudo
Hi @Nathan Maynes Im using GetFile and SplitFile to take the rows of data from the file (which gives me a flowfile consisting of 30 features) then set a control rate of 1 flowfile per second before sending it to PutTCP processor. I am trying to emulate a device sending data in real time to my server on localhost, where i perform processing on the incoming flowfile. But the server gives an error which seems to be related to the format of the incoming flowfile.
... View more
01-30-2018
12:12 PM
Hi @Nathan Maynes. The list is coming from a file. I am sending it to a python server on the local machine via a PutTcp processor. But every now and then the python server gives an error.
... View more
01-30-2018
11:56 AM
Hi, I am trying to send a list to a specific port with NiFi. However, i am getting issues when the server receives the list. I think an option is to serialize the list before sending it. Is this possible with Nifi or should i use my own code with ExecuteStreamCommand processor? Thank you
... View more
Labels:
- Labels:
-
Apache NiFi
01-16-2018
01:22 PM
Hi, i am working on a project and i am wondering if minifi can dynamically transmit data to a different url without restarting? Currently i am using a remote process group for bidirectional communication between the minifi device and nifi server. Thank you.
... View more
Labels:
- Labels:
-
Apache MiNiFi
-
Apache NiFi
12-16-2017
02:17 PM
Hi @Ed Berezitsky. I assign the output of ListenTCP an attribute called "request" and pass it into my python code via command line argument as shown here: C:\Users\r.young\threshold-1.py;${request} However, i get an error saying list index sys.argv[1] out of range. I know the ${request} is correct because when i send it directly to a putTCP processor, i can view its content, the issue i am having is trying to pass it correctly into my code. Thank you
... View more
12-10-2017
02:45 PM
Hi, I am trying to pass output from a ListenTCP processor into a python code. I have tried passing it in as a command line argument using an ExecuteStreamCommand processor, but had no luck this way, because i had to assign the output an attribute name to pass it as an argument. Is there an alternative way? Thank you.
... View more
Labels:
- Labels:
-
Apache NiFi
08-23-2017
12:32 PM
Hi @Wynner. I have it kind of working with a python script and command line arguments but i thought there might have been a more straight forward way of doing it. Thank you for the reply.
... View more
08-18-2017
10:01 AM
Hi, i have a dataflow in a RPG that ingests data and assigns attributes to each feature in the flowfile as shown below, before transmitting to the main NiFi server. What i am trying to do is make a request from the main NiFi server for a particular attribute "P1" for example. So if i made a request for P1, (via python script or listenTCP) only P1 would be transmitted from the RPG to the main NiFi server. Is there a way of doing this, thank you.
... View more
Labels:
- Labels:
-
Apache NiFi
07-05-2017
02:27 PM
Hi, i have been following some online examples in trying to build a model. I am using a csv data set, Below is a snippet of the headings and some of the data:
TrialID ObsNum IsAlert P1 P2 P3 P4 P5 P6 P7 P8 E1 E2 0 0 138.4294 10.9435 1000 60 0.302277 508 118.11 0 0 0 0 1 138.3609 15.3212 1000 600.302277 508 118.11 0 0 0 The third column, IsAlert is the ground truth This is the code i have been trying, amongst some others. val training = sc.textFile("hdfs:///ford/fordTrain.csv")
val header = training.first
val inferSchema = true
val lr = new LogisticRegression() .setMaxIter(10) .setRegParam(0.3) .setElasticNetParam(0.8)
// Fit the model
val lrModel = lr.fit(training)
// Print the coefficients and intercept for logistic regression
println(s"Coefficients: ${lrModel.coefficients} Intercept: ${lrModel.intercept}")
// We can also use the multinomial family for binary classificationval
mlr = new LogisticRegression() .setMaxIter(10) .setRegParam(0.3) .setElasticNetParam(0.8) .setFamily("multinomial")
val mlrModel = mlr.fit(training)
// Print the coefficients and intercepts for logistic regression with multinomial family
println(s"Multinomial coefficients: ${mlrModel.coefficientMatrix}")
println(s"Multinomial intercepts: ${mlrModel.interceptVector}")
This is the error i am recieving
import org.apache.spark.sql.types.{StructType, StructField, StringType}
training: org.apache.spark.rdd.RDD[String] = hdfs:///ford/fordTrain.csv MapPartitionsRDD[7] at textFile at <console>:188
header: String = TrialID,ObsNum,IsAlert,P1,P2,P3,P4,P5,P6,P7,P8,E1,E2,E3,E4,E5,E6,E7,E8,E9,E10,E11,V1,V2,V3,V4,V5,V6,V7,V8,V9,V10,V11
inferSchema: Boolean = true
lr: org.apache.spark.ml.classification.LogisticRegression = logreg_1049bed7e9a0
<console>:192: error: type mismatch;
found : org.apache.spark.rdd.RDD[String]
required: org.apache.spark.sql.DataFrame
val lrModel = lr.fit(training)
^
I would be grateful for any help, thank you
... View more
Labels:
- Labels:
-
Apache Spark
-
Apache Zeppelin
05-25-2017
01:55 PM
i just killed the processes in task manager
... View more
05-25-2017
01:09 PM
Hi @apsaltis
I figured out where i went wrong. The config.sh transform command wasnt working as i was on a windows machine. I used config.bat and its working fine now
... View more
05-25-2017
11:16 AM
Below is the config.yml file in minifi. There is no mention of processors there, maybe ive done something wrong # Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the \"License\"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an \"AS IS\" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
MiNiFi Config Version: 3
Flow Controller:
name: MiNiFi Flow
comment: ''
Core Properties:
flow controller graceful shutdown period: 10 sec
flow service write delay interval: 500 ms
administrative yield duration: 30 sec
bored yield duration: 10 millis
max concurrent threads: 1
FlowFile Repository:
partitions: 256
checkpoint interval: 2 mins
always sync: false
Swap:
threshold: 20000
in period: 5 sec
in threads: 1
out period: 5 sec
out threads: 4
Content Repository:
content claim max appendable size: 10 MB
content claim max flow files: 100
always sync: false
Provenance Repository:
provenance rollover time: 1 min
Component Status Repository:
buffer size: 1440
snapshot frequency: 1 min
Security Properties:
keystore: ''
keystore type: ''
keystore password: ''
key password: ''
truststore: ''
truststore type: ''
truststore password: ''
ssl protocol: ''
Sensitive Props:
key: ''
algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL
provider: BC
Processors: []
Process Groups: []
Funnels: []
Connections: []
Remote Process Groups: []
NiFi Properties Overrides: {}
... View more
05-25-2017
09:32 AM
1 Kudo
Hi @apsaltis
I have followed this tutorial and it works great. One thing i am unsure about. Once the template is downloaded onto minifi and minifi is running, in theory i should be able to delete the GenerateFlowfileProcessor and RPG from the canvas and NiFI still receive data from Minifi. But when i do this, NiFi doesnt receive anything? How is this happening? Thank You
... View more
05-25-2017
08:57 AM
Hi, that command returns the following error 'nifi.bat' is not recognized as an internal or external command,
operable program or batch file.
... View more
05-24-2017
01:24 PM
Hi, staus-nifi.bat shows NiFi is running. There is also java processes showing in task manager. NiFi is working fine, i just want to shut it down properly. I tried bin\nifi.sh stop but that doesnt work.
... View more
05-24-2017
12:56 PM
Hi, I didnt stop NiFi with the usual control+C command, when i turned my laptop back on NiFi was still running. How do i Shut it down? Thanks
... View more
Labels:
- Labels:
-
Apache NiFi
03-23-2017
12:23 PM
Hi @Matt Clarke. I have been using nifi fine in a virtual machine. Recently i have it installed on my windows 10 host but i keep getting the same error as above even though i have configured the nifi.properties file as follows. I am just using localhost at the minute as i have minifi also installed for testing. # Site to Site properties
nifi.remote.input.host=localhost
nifi.remote.input.secure=false
nifi.remote.input.socket.port=10000
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec
# web properties #
nifi.web.war.directory=./lib
nifi.web.http.host=
nifi.web.http.port=9191
nifi.web.https.host=
nifi.web.https.port=
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
Is there any other configurations to be changed? Thank you
... View more
02-28-2017
07:44 PM
Hi @Matt Burgess yes its the latter. No issue with receiving the whole JSON, i was just trying to minimize the response if possible.
... View more
02-28-2017
05:40 PM
Hi, currently i am using InvokeHTTP to retrieve a JSON file from a weather website. Then i use EvaluateJsonPath to take the attributes i need from the json file. Is it possible to just recieve the attributes from a url instead of receiving the whole json file? Thank you
... View more
Labels:
- Labels:
-
Apache NiFi
02-28-2017
04:06 PM
hi @bhagan i want my output to become an attribute, but im having trouble assigning it an attribute with update attribute processor. What do i assign as my value? ${output stream} doesnt work
... View more
02-27-2017
02:08 PM
Hi, i have a custom program producing an integer output when using an ExecuteScriptCommand. I am trying to assign the integer an attribute with UpdateAttribute processor but its not working? How could i go about this, thank you
... View more
Labels:
- Labels:
-
Apache NiFi
02-16-2017
12:21 PM
Hi, that makes sense. I used ExtractText and it worked, thanks for the help
... View more
02-15-2017
06:54 PM
Hi @Bryan Bende. Thanks for the reply. I added a logAttribute and it shows there is no value for latitude or longitude. i dont know why its not working. The data i am using is from the data simulator in the trucking app. The previous processor routes the truck_geo_events to the updatettribute processor.
longitude {line:getDelimitedField(9,'|')}
2017-02-15 18:49:56.779|truck_geo_event|92|10|George Vetticaden|1594289134|Memphis to Little Rock Route 2|Normal|34.8|-92.09|1|
This looks like it should work though?
... View more
02-14-2017
06:23 PM
Hi, i have an UpdateAttribute processor that creates 2 attributes longitude and latitude as follows longitude {line:getDelimitedField(9,'|')} i am trying to add them into a url in the following InvokeHTTP processor with the following configuration but its not working http://api.met.no/weatherapi/locationforecast/1.9/?lat=${latitude};lon=${longitude}
Is this possible or have i gone about it incorrectly. The URL works when i enter in the longitude and latitude figures manually
Thank you
... View more
Labels:
- Labels:
-
Apache NiFi
02-08-2017
11:41 AM
Hi, is it possible to send data to a specific url using a processor, and retrieve information back based on the data sent? For example, send a password to a website that returns the strength of the password. Thank you
... View more
02-03-2017
12:51 PM
@Pierre Villard yep, that did it, thanks for the help.
... View more