Member since
06-27-2016
19
Posts
1
Kudos Received
0
Solutions
09-05-2017
05:50 AM
@David Sheard I am facing the same issue, please let me know in case you were able to resolve it.
... View more
09-05-2017
05:50 AM
@nedox nedox Were you able to resolve this issue?
... View more
11-21-2016
01:25 PM
@Dima Avramov Yes, referred this pull request https://github.com/apache/spark/pull/13359 We need to write a custom Teradata dialect and force String type to varchar. The same was done when using Oracle till few months back http://stackoverflow.com/questions/31287182/writing-to-oracle-database-using-apache-spark-1-4-0 Let me know if this works for you.
... View more
11-17-2016
11:37 AM
Hi, I am reading a table in Teradata into Spark and encoding to a custom POJO. There are some fields declared as String. When writing back to Teradata I get the following error:
Exception in thread "main" java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.26] [Error 3706] [SQLState 42000] Syntax error: Data Type "TEXT" does not match a Defined Type name.
String is getting mapped to TEXT instead of varchar. I have tried changing the Teradata JDBC versions yet get the same error. Anyone knows how to force it to save as varchar instead of TEXT? Thanks.
... View more
Labels:
07-27-2016
01:47 PM
Hi @Andrew Grande, I tried your suggestion and added the JVM settings namely -Dhttps.proxyHost, -Dhttps.proxyPort, -Dhttp.proxyUser and -Dhttp.proxyPassword to bootstrap.conf (I am trying to set proxy with username/password for streaming Tweets using the GetTwitterProcessor) However even after restarting NiFi I am not able to connect successfully. Can you please provide an alternative solution? Thanks.
... View more
07-21-2016
05:44 AM
Can NiFi store current session state variables when being exposed as a web service using HandleHttpRequest-HandleHttpResponse?
Does the State Manager store state application or session wise?
... View more
Labels:
07-08-2016
05:58 AM
Hi @Timothy Spann, Can you pls tell me why Storm was chosen as the first framework for HDF? Is it because it offers a real-time streaming environment, as opposed to batch streaming by Spark? Thanks.
... View more
07-05-2016
12:59 PM
I read that setting MAHOUT_LOCAL="TRUE" can set Mahout to run in local mode, implying that I can read files from the local system instead of HDFS. I want to know if running the mahout spark-itemsimilarity job will run in a distributed fashion in the Spark cluster or will it be a single machine execution? I have set the --master option to the Spark Master URL.
PS I want to read from the local file system, not from HDFS. Also, I want to utilize the complete Spark cluster.
... View more
Labels:
07-01-2016
08:00 AM
Thanks for the reply @Jobin George, can you tell me how to send the results from Spark back to NiFi? There is a NiFiReceiver but no NiFiSender. Does the former have a method to send back the results? (I currently use the Java version)
Thanks.
... View more
07-01-2016
06:51 AM
Thank you for the suggestion @Jobin George, you are right. I have 4 machines, one Namenode, one NCM, and two Datanodes/NiFi nodes. I guess that is why I did not face the error. Apologies for the comments.
Is this setup of 4 machines with shared components a good approach, or do you suggest having separate machines for the NiFi nodes? I will not be dealing with too much overhead at the datanodes, only when there is a need for nightly model re-training and during model predictions (please also have a look at my query in your NiFi + Spark : Feeding Data to Spark Streaming thread) Thanks for your time and patience 🙂
... View more
07-01-2016
06:34 AM
The NiFi cluster NCM url 🙂
... View more
07-01-2016
06:29 AM
I do not have any issues @Jobin George, I am able to transfer data from NiFi into HDFS from a browser in the Namenode, by referencing the path of the configurations files inside Hadoop directory from Namenode. I want to know if by this method, NiFi is able to run in a full clustered mode(since the config files are not copied to other NiFi nodes) or does it internally run as a single node setup. Thanks.
... View more
07-01-2016
05:55 AM
Hi @Jobin George,
I have a query regarding your answer. I have a 3 node NiFi cluster setup and a 3 node HDP setup. Though I faced the same issue accessing the UI from NCM, I did not get any error when I accessed it from a browser in the Hadoop Namenode. I referenced the config files from inside the Namenode and data was transferred from NiFi to HDFS directory successfully. It may not be a good approach to access NiFi from the Namenode in production, but for experimentation and learning purposes can you pls try the above and let me know if it utilizes all the NiFi nodes or is it running in a single node(which defeats the use of the cluster) Also, if the above method does work, any suggestions to suit the production environment? Thanks.
... View more
07-01-2016
05:34 AM
@mclark can you give us an approximate month when NiFi 1.0 will be available to the community? Thanks.
... View more
06-30-2016
11:09 AM
1 Kudo
Hi, I have tried building a web service with NiFi and am able to get the incoming requests and pass it to Spark/Storm. Assuming that I compute the prediction inside Spark, I wish to know how to send back the score/result as response to NiFi. If that is currently not possible, what are the chances of creating a custom processor in R to predict the scores and pass it on as response? Thanks.
... View more
06-30-2016
10:54 AM
Thank you very much for the reply. If its the case of priority then I understand. 🙂 Sure, I will open the ML query as a separate one.
... View more
06-30-2016
10:39 AM
Hi @Benjamin Leonhardi, Thanks for replying. You are right, there is a connector to receive the data from NiFi to Spark. I had tried it by transferring the data to Spark through an output port and it works pretty well 🙂 My question is more on why Hortonworks decided not to include it. I am also curious on the Machine Learning integration. Please share if there are any good ways to accept requests from NiFi, make predictions using a trained classifier and finally give back the scores as response. Here again, NiFi can accept requests and give back a response, but I am not able to give it to a classifier and get the scores back. Thanks again.
... View more
06-30-2016
10:28 AM
HDF ships NiFi,Kafka,Zookeeper and Storm as default components. Why not Spark? I understand that Spark is not meant for streaming, rather better suited for micro-batching, but any additional reasons?
Thanks.
... View more
Labels:
06-27-2016
09:59 AM
Hi @Jobin George, Thanks for the tutorial! I am connecting to Spark from Nifi using the Output Port. I tried an example - sending a HTTP POST request using the HandleHttpRequest processor with random JSON message which is processed by Spark (code in Java) and is ultimately stored in a MongoDB collection. As a next step, I am going to send a real-world JSON request (let's say IoT data) and predict using a model (which I had already trained using MLLib). I want to predict the score and *return the result* back to Nifi's HandleHttpResponse processor as a response. I am not able to connect an output relationship from the Output Port to the Response processor. Is there another way to send the response (from Spark side) ? Thanks, Sai
... View more