About srowen

srowen · ‎05-17-2016

Yes, that sounds right, though I confess I haven't tried that myself. Others here may have better suggestions.

srowen · ‎03-24-2016

If you mean persisting in memory, then one reason of course is not having enough memory to fully cache it. If you have enough memory, then the key is to see that it doesn't actually wait for partitions to cache to compute the result. If you run several times you should find each subsequent run finds more of the partitions cached, and so runs faster, but each will complete without waiting for everything to serialize to the cache. I'm not sure that's an issue at all. Your first run necessarily can't use the cache. It would be slower if you waited for the thing to cache fully before answering.

srowen · ‎03-23-2016

Are you maybe looking for the Anaconda parcel for CDH? http://know.continuum.io/anaconda-for-cloudera.html That gives you a fixed distribution of Python that supports Pyspark and probably your other use cases on the cluster. This is what virtualenvs can help with in any event.

srowen · ‎03-20-2016

The map() will execute on the cluster. There are some problems with this code, but the most immediate one is that you are using inner classes, which are non-static in Java, and you are creating them in an instance method. They have a reference to your containing class which presumably is unintentional, and which contains non-serializable elements. Put this in a static method.

srowen · ‎03-19-2016

It depends on what your code does. Right now this code just shows a process that collects some data to the driver, calls your naive Bayes-related function on the driver, and collects some other data to the driver. This code needs to call distributed operations if you want it to be distributed. What that is, I couldn't say, since I'm not sure what you're trying to invoke.

srowen · ‎03-19-2016

word2vec is just a means of translating bags of items to a vector space representation. I myself don't call that NLP per se but it is used to make feature vectors from text. NLP to me is more like stemming and sentiment analysis. For this you'd be calling to third-party libraries, like the Stanford NLP library, or building your own NLP processes on top of generic implementations of, say, LDA in Spark.

srowen · ‎03-19-2016

Yes, you can call any package you like from a function. But the packages will not use Spark. They're executing on one node as usual. This is just fine if you want to do some local transformation with a package, but not if you're expecting it will parallelize a decision forest or something. You can call through to some of the implementations in Spark MLlib for that.

srowen · ‎03-19-2016

Spark's package is an API to Spark. It does not use, nor is it used by, CRAN packages that provide, say, some kind of statistical function. You can't use the "maxEntropy" package you're referring to with SparkR.

srowen · ‎03-19-2016

As I say, you are not calling foreach. You are calling foreachRDD. That executes on the driver and is supposed to. It defines what functions are invoked on each RDD. One of those functions can be foreach, but that's not what you're calling. The function you invoke for distributed operations must be serializable.

srowen · ‎03-18-2016

foreach is an action, not a transformation. It is not true that transformation execute on the driver; they are distributed. foreachRDD executes a function on the driver. I'm not clear therefore what you are doing vs experiencing.

srowen · ‎03-18-2016

Spark has no particular support for NLP, no. You can use third party libraries for this.

srowen · ‎03-18-2016

Spark does not parallelize or affect CRAN packages in any way. You can't execute regular R code in a distributed way. SparkR is not a distribution of R. It is an API for using Spark from R. SparkR provides no CRAN libraries at all. It is just an interface to write your own Spark programs in R.

srowen · ‎03-14-2016

CDH 5.6 is already released and does not support SparkR; although I can't promise I know it *won't* be in CDH5.7, I don't believe it is. It's actually in the tree (https://github.com/cloudera/spark/tree/cdh5-1.5.0_5.6.0/R) and any similarly-versioned Spark distribution could probably be built such that its SparkR works with CDH. You'd have to build it manually. The big issue is that yeah you'd also have to manually install R and all the supporting packages on your cluster.

srowen · ‎03-14-2016

This is a network config question, not CDH or Spark. They can communicate if there is a path between them. You normally assign any node you want to run Spark from as a Spark gateway node (you don't otherwise "install Spark" per se). This can be separate from the nodes in the YARN node/resource manager roles, but usually is *not* separate. That is, making the driver remote is not necessarily a good idea.

srowen · ‎03-11-2016

You can look this up online. Yes, 7.1 is supported. http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_req_supported_versions.html

srowen · ‎03-10-2016

Yes, actions invoked serially in the driver will execute serially. That's your second example. You can execute them in parallel in the driver with a little bit of code, and that should also work perfectly fine. However I think, or thought, that an RDD with two parents would have them both evaluated at the same time if possible. You can click through to the job's DAG visualization to get a sense of what it thinks it is doing and in what order. I'd have to also run a little test to confirm or deny whether it works this way.

srowen · ‎03-09-2016

It should be able to parallelize these but it would also require enough executor slots to cover all of the tasks. So if # partitions for both is more than your free slots, I don't think it would schedule a second stage. Have a look at the DAG in the UI to confirm the sequence of ops is actually parallelizable according to Spark.

srowen · ‎03-09-2016

That's not what it says; it say they just aren't supported, typically because they're not "supported" in Spark either (e.g. experimental API). Supported != doesn't work, just means you can't file a support ticket for it. CDH 5.6 = Spark 1.5 + patches, meaning it's like 1.5.2 likely with a slightly different set of maintenance patches. It might not have unimportant ones that maybe shouldn't be in a maintenance release, or might have a critical one that was created after 1.5.2. Generally speaking there are no other differences; it's just upstream Spark with some tinkering with versions to make it integrate with other Hadoop components correctly. The exception is SparkR, which isn't even shipped, partly because CDH can't ship R itself.

srowen · ‎03-08-2016

If the brokers are entirely unavailable, I believe it would fail the job, yes. The job can't read any data anyway, so it can't proceed anyway. I think you'd just have to restart the job if your cluster were offline completely (or try to negotiate more reliability from the other team if that's really the problem?)

srowen · ‎03-07-2016

It includes an implementation of classification using random decision forests. Decision forests actually support both categorical and numeric features. However, for text classification, you're correct that you typically transform your text into numeric vectors via TF-IDF first. This is something you'd have to do separately. Yes, the dimensionality is high. Decision forests can be fine with this, but, they're not the most natural choice for text classification. You may see what I mean that Oryx is not a tool for classification, but a tool for productionizing, which happens to have an implementation of a classifier. In 2.x, you also have an implementation of decision forests, and also don't have magic TF-IDF built in or anything. However the architecture is much more supportive of putting your own Spark-based pipeline and model build into the framework. 1.x did not support this.

srowen · ‎03-06-2016

Yes, though I would describe Oryx as support for productionizing some kind of learning system. Just making a model is something you should do with other tools whose purpose is to build models. Oryx 1 is not exactly deprecated, but Oryx 2 is the only version in active development, and I'd really encourage you to look there. The good news is that it's a lot easier in 2.x to reuse a model building process you created in, say, Spark. In 1.x it's not possible.

srowen · ‎03-04-2016

You can always look it up at http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_vd_cdh_package_tarball.html While individual components might have a version in some kind of UI, not all even have a UI. They don't have some consistent programmatic identifier. The next-best thing is to grep the names of JAR files in the distribution, if you really had to do it programmatically.

srowen · ‎03-03-2016

scalac is not part of Spark; it is the Scala compiler. It's not clear what you mean by "Spark terminal", but the Spark shell is not a command line, and not something related to scalac. It is a shell where you type code, like the Scala shell.

srowen · ‎03-03-2016

I don't think the JIRA you linked is related and I'm not clear from what you posted that there is a problem. What result do you expect? With that kind of info, you'd need to contact support.

srowen · ‎03-02-2016

http://oryx.io/ describes the architecture and http://oryx.io/docs/developer.html says more about the project module structure. The javadoc for individual classes describes the component input/output; you can also inspect the code or just watch the messages on the queue to see it in action.

srowen · ‎02-29-2016

The error points to the problem -- you have perhaps plenty of memory but not enough permgen space in the JVM. Try something like -XX:MaxPermSize=2g in your JVM options to executors

srowen · ‎02-25-2016

No, the example is compiled along with everything else and has always passed ... https://travis-ci.org/OryxProject/oryx/branches What are you seeing?

srowen · ‎02-22-2016

"Not supported" means you can't file support tickets for it. It's shipped and works though.

srowen · ‎02-22-2016

Yes, always has been. You can inspect the assembly JAR or just try using it in the shell -- did you try it? CDH ships all of Spark in general. The only thing I think isn't included in SparkR's R bits.

srowen · ‎02-12-2016

Oops, it really should have always been 1.3.0-cdhx.x.x since that is the version in the Maven artifacts. In any event, 1.5.0-cdh5.5.1 is a valid version string that this third party library should not reject.

Online	Offline
Last Visited	‎02-06-2015 02:06 PM

Date Registered	‎07-29-2013 08:58 AM
Last Visited	‎02-06-2015 02:06 PM
Total Messages Posted	366
Total Kudos Received	62

Re: CDH 5.6

Re: How to use Oryx 1 to detect spam email

Re: Spark program in eclipse

Re: Graphx in latest CDH

Re: Maturity ORYX

Re: Idle Spark Shells

Re: RDD not fully persisting

Re: Conda env

Re: kafka-spark mutliple worker nodes

Re: kafka-spark mutliple worker nodes

Re: Spark NLP provision

Re: Spark - R (cran) integration

Re: Spark - R (cran) integration

Re: kafka-spark mutliple worker nodes

Re: kafka-spark mutliple worker nodes

Re: Spark NLP provision

Re: Spark - R (cran) integration

Re: SparkR in CDH 5.5?

Re: Spark-YARN connectivity in different machines

Re: Curios To know is CDH5 fully supports CentOS7

Re: Getting Spark stages to run in parallel inside...

Re: Getting Spark stages to run in parallel inside...

Re: CDH 5.6

Re: Spark Streaming application shutdown when Kafk...

Re: How to use Oryx 1 to detect spam email

Re: How to use Oryx 1 to detect spam email

Re: CDH components version numbers

Re: Scalac not working in spark

Re: CDH Spark window functions return wrong result...

Re: How does speed layer updates model?

Re: Spark program in eclipse

Re: How does speed layer updates model?

Re: Graphx in latest CDH

Re: Graphx in latest CDH

Re: SAP HANA VORA fails with SPARK 1.5 CDH 5.5