@nedox nedox Were you able to resolve this issue? ... View more

@Dima Avramov Yes, referred this pull request https://github.com/apache/spark/pull/13359 We need to write a custom Teradata dialect and force String type to varchar. The same was done when using Oracle till few months back http://stackoverflow.com/questions/31287182/writing-to-oracle-database-using-apache-spark-1-4-0 Let me know if this works for you. ... View more

Hi @Andrew Grande, I tried your suggestion and added the JVM settings namely -Dhttps.proxyHost, -Dhttps.proxyPort, -Dhttp.proxyUser and -Dhttp.proxyPassword to bootstrap.conf (I am trying to set proxy with username/password for streaming Tweets using the GetTwitterProcessor) However even after restarting NiFi I am not able to connect successfully. Can you please provide an alternative solution? Thanks. ... View more

Thanks for the reply @Jobin George, can you tell me how to send the results from Spark back to NiFi? There is a NiFiReceiver but no NiFiSender. Does the former have a method to send back the results? (I currently use the Java version) Thanks. ... View more

Thank you for the suggestion @Jobin George, you are right. I have 4 machines, one Namenode, one NCM, and two Datanodes/NiFi nodes. I guess that is why I did not face the error. Apologies for the comments. Is this setup of 4 machines with shared components a good approach, or do you suggest having separate machines for the NiFi nodes? I will not be dealing with too much overhead at the datanodes, only when there is a need for nightly model re-training and during model predictions (please also have a look at my query in your NiFi + Spark : Feeding Data to Spark Streaming thread) Thanks for your time and patience 🙂 ... View more

The NiFi cluster NCM url 🙂 ... View more

I do not have any issues @Jobin George, I am able to transfer data from NiFi into HDFS from a browser in the Namenode, by referencing the path of the configurations files inside Hadoop directory from Namenode. I want to know if by this method, NiFi is able to run in a full clustered mode(since the config files are not copied to other NiFi nodes) or does it internally run as a single node setup. Thanks. ... View more

Hi @Jobin George, I have a query regarding your answer. I have a 3 node NiFi cluster setup and a 3 node HDP setup. Though I faced the same issue accessing the UI from NCM, I did not get any error when I accessed it from a browser in the Hadoop Namenode. I referenced the config files from inside the Namenode and data was transferred from NiFi to HDFS directory successfully. It may not be a good approach to access NiFi from the Namenode in production, but for experimentation and learning purposes can you pls try the above and let me know if it utilizes all the NiFi nodes or is it running in a single node(which defeats the use of the cluster) Also, if the above method does work, any suggestions to suit the production environment? Thanks. ... View more

Hi, I have tried building a web service with NiFi and am able to get the incoming requests and pass it to Spark/Storm. Assuming that I compute the prediction inside Spark, I wish to know how to send back the score/result as response to NiFi. If that is currently not possible, what are the chances of creating a custom processor in R to predict the scores and pass it on as response? Thanks. ... View more

Hi @Jobin George, Thanks for the tutorial! I am connecting to Spark from Nifi using the Output Port. I tried an example - sending a HTTP POST request using the HandleHttpRequest processor with random JSON message which is processed by Spark (code in Java) and is ultimately stored in a MongoDB collection. As a next step, I am going to send a real-world JSON request (let's say IoT data) and predict using a model (which I had already trained using MLLib). I want to predict the score and *return the result* back to Nifi's HandleHttpResponse processor as a response. I am not able to connect an output relationship from the Output Port to the Response processor. Is there another way to send the response (from Spark side) ? Thanks, Sai ... View more