Created on 04-30-2016 02:53 AM - edited 09-16-2022 03:16 AM
Hello guys,
First, i want to make a project with Apache Nifi, but there is some problem that i face it now.
1. Can i make Machine Learning Calculation in Apache Nifi ?
I know that i must use Mahout and Spark, but i dont find Mahout and Spark Processor in Apache Nifi. What should i do ?
Can you suggest me another idea.
2. I want to make web service with Apache Nifi ?
First time, i use ListenHTTP Processor, but the text that i send become Flow File. How can i convert it to text so i can put it in GetTwitter Processor ?
Thanks for reply 🙂
Created 04-30-2016 03:03 AM
The more traditional approach in this situation is to use NiFi to read the incoming data and then add a NiFi processor to dump the data from the NiFi queue to either Storm or in your case SparkStreaming. Now you can build a Spark ML model test it and run it in Spark. You can push logic into NiFi but an ML model inside NiFi is overkill.
Created 04-30-2016 03:05 AM
Vadim and I at hortonworks have a packaged demo of exactly what I described above on github. data streaming through NiFi into a spark model at this link https://github.com/vakshorton
Created 05-02-2016 02:57 PM
@Kirk Haslbeck Thanks, i will try it, one of them as fast as possible
Created 12-09-2016 07:44 PM
Hi @Kirk Haslbeck i have tried some of the demo packages you mentioned below but keep having issues with connecting to nifi servlet etc, are the demos compatible with 2.5? And i also get the following error.
"An internal system exception occurred: Could not find service for component, componentName=NIFI_MASTER, clusterName=Sandbox, stackInfo=HDP-2.5"
Am i missing something? thanks
Created 04-30-2016 03:08 AM
Created 05-02-2016 03:04 PM
@Ravi Mutyala Hmmm..., I dont know if there are more tutorial about Apache Nifi in this forum than Hortonworks Tutorial. Better to check it, and thanks a lot.
Created 04-30-2016 09:04 AM
As suggested by Ravi, have a look at :
It will show you how to connect NiFi and Spark if you want to use this kind of architecture to perform machine learning.
Regarding your second question, GetTwitter does not accept ingoing connection. What are you trying to achieve?
Created 05-02-2016 03:01 PM
@Pierre Villard I want to send Twitter username from smartphone to HDP. When ListenHTTP get the data, it will send to GetTwitter to get some tweet according username. After that, i will use spark to calculate Machine Learning Algorithm to predict user personality.
Created 05-02-2016 03:08 PM
As I said, GetTwitter does not accept incoming relationship. You won't be able to connect ListenHTTP to GetTwitter. Your best option is to "build" your own flow to request Twitter data (example: https://pierrevillard.com/2016/04/12/oauth-1-0a-with-apache-nifi-twitter-api-example/), or to write your own processor using the existing one as an example.
Created on 04-30-2016 03:10 PM - edited 08-19-2019 03:14 AM
What are you trying to build is what we call Connected Data Platform at Hortonworks. You need to understand that you have two types of workloads/requirements and you need to use HDF and HDP jointly.
So you will have to use HDP to construct the model, HDF to get and transform the data, and finally a combination of HDF/HDP to apply the model and make the prediction.
To build a web service with NiFi you need to use several processors: one to listen to incoming requests, one or several processors to implement your logic (transformation, extraction, etc), one to publish the result. You can check this page that contains several data flow examples. The "Hello_NiFi_Web_Service.xml" gives an example on how to do it.
https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates
Created 06-30-2016 11:09 AM
Hi,
I have tried building a web service with NiFi and am able to get the incoming requests and pass it to Spark/Storm. Assuming that I compute the prediction inside Spark, I wish to know how to send back the score/result as response to NiFi.
If that is currently not possible, what are the chances of creating a custom processor in R to predict the scores and pass it on as response?
Thanks.
Created 07-21-2017 08:13 AM
Can we automate a R script in nifi.
Created 10-08-2017 10:33 PM
Call Spark: https://community.hortonworks.com/articles/73828/submitting-spark-jobs-from-apache-nifi-using-livy.h...
Or call Machine learning from NiFi or write Machine Learning or Deep Learning components in nifi
https://community.hortonworks.com/articles/80418/open-nlp-example-apache-nifi-processor.html
Created 10-08-2017 10:34 PM
easy to integrate NiFi -> Kafka -> Spark or Storm or Flink or APEX
Also NiFi -> S2s -> Spark / Flink / ...