About srowen

srowen · ‎03-29-2015

Yes, that's right.

srowen · ‎03-29-2015

Yes, it becomes a new "row" in Y. The candidate filter is something else. This line of code is like a callback notifying the implementation that a new item exists. New users also cause a new row in X, but there is no equivalent 'candidate filter' for users because the same types of operations (recommend, etc.) are not supported for users.

srowen · ‎03-26-2015

Hm, how do you compile your app? Usually you create a Maven or SBT project to declare its dependencies, which should include a "provided" dependency on the same version of Spark as is on your cluster. How do you submit your application? spark-submit? you are submitting a JAR to run your app, right?

srowen · ‎03-25-2015

I mean, do you build your app with a dependency on Spark, and if so what version, and, have you marked it as 'provided' so as not to be included the JAR you submit?

srowen · ‎03-25-2015

This generally means you're mixing two versions of Spark somehow. Are you sure your app isn't also trying to bundle Spark? are you using the CDH Spark, and not your own compiled version?

srowen · ‎03-25-2015

To add a little color, yes you can do that, although the CLASSPATH intentionally does not include Hive, since as I understand, Spark doesn't work with the later versions of Hive that CDH 5.3 and beyond use. It still may work enough to do what you need, so, have at it. But you may hit some incompatibilities.

srowen · ‎03-18-2015

Looks good although I would recommend closing the statement and connection too. Also, you're executing an update for every datum. JDBC as an addBatch / executeBatch interface too I think? might be faster.

srowen · ‎03-16-2015

Yes, perfectly possible. It's not specific to Spark Streaming or even Spark; you'd just use foreachPartition to create and execute a SQL statement via JDBC over a batch of records. The code is just normal JDBC code.

srowen · ‎03-15-2015

Although that thread sounds similar, I don't think it's the same thing. Failing to bind is not a failure to connect to a remote host. It means the local host didn't allow the process to listen on a port. The two most likely explanations are: - an old process is still listening on that port, or at least, another still-running process is - you appear to be binding to a non-routable address (192.168.x.x) This might be OK but worth double-checking

srowen · ‎03-15-2015

As you can see, the problem is that the receiver can't bind to its assigned address. Is there any networking-related restriction in place that would prevent this? is this the port you intended?

Online	Offline
Last Visited	‎02-13-2018 12:34 PM

Member Since	‎08-11-2014 09:17 AM
Last Visited	‎02-13-2018 12:34 PM
Posts	481
Kudos received	87

Cloudera Community

Re: Own code editor in CDSW?

Re: error using Pandas within PySpark transformati...

Re: Does CDSW need to be part of the cluster?

Re: Local Data combined with HDFS

Re: Where can I find Oryx 1.x releases (or GitHub)

Re: Retrieve and modify latent feature vectors on ...

Re: Retrieve and modify latent feature vectors on ...

Re: Spark (Standalone) error local class incompati...

Re: Spark (Standalone) error local class incompati...

Re: Spark (Standalone) error local class incompati...

Re: spark sql is not working on CDH5.3

Re: Spark Streaming save output to mysql DB

Re: Spark Streaming save output to mysql DB

Re: Spark Streaming Fails on Cluster mode ( Flume ...

Re: Spark Streaming Fails on Cluster mode ( Flume ...