To add a little color, yes you can do that, although the CLASSPATH intentionally does not include Hive, since as I understand, Spark doesn't work with the later versions of Hive that CDH 5.3 and beyond use. It still may work enough to do what you need, so, have at it. But you may hit some incompatibilities. ... View more

Looks good although I would recommend closing the statement and connection too. Also, you're executing an update for every datum. JDBC as an addBatch / executeBatch interface too I think? might be faster. ... View more

As you can see, the problem is that the receiver can't bind to its assigned address. Is there any networking-related restriction in place that would prevent this? is this the port you intended? ... View more

I don't think there's anything special to know, beyond what's documented in the RHadoop subprojects. So it's not something that we ship, support or document separately. I have set up the rhadoop libraries with CDH and it's straightforward.   It's really a set of client side libraries that you install into *R*, not *Hadoop*. However to run rmr2 you will need R installed locally on all of your Hadoop cluster nodes, since it will run MapReduce jobs that execute R scripts.   I recall that you have to install a bunch of other R packages before installing the rhdfs/rhbase/plyrmr libraries, and I found this in my notes as the set of prerequisites:     install.packages(c("Rcpp", "RJSONIO", "bitops", "digest", "functional", "reshape2", "stringr", "plyr", "caTools", " rJava ", "dplyr", "R.methodsS3", "Hmisc")) ... View more

You can also just add the HIve jars to your app classpath. The catch here is that Spark doesn't quite support the later version of Hive in CDH. This might work for what you're trying to do, but if you build your own, you're building for a slightly different version of Hive than you run here. ... View more

In bash for example: unset VARIABLE   ... View more

It's just an env variable; you can always "export VARIABLE=VALUE" in the shell. But this message is not an error. In fact you want to see this if you intend to run on a cluster. ... View more

Mahout is already shipped with CDH. It's not something you really install; it's a library. Can you say any more about the problem you are facing? ... View more

What is null? I don't see you using a broadcast fariable in a closure here. You just put one in a static member, which isn't going to work. ... View more

There is no absolute minimum. I would imagine that, for reasoanbly sized problems, you'd want to allocate 1GB of memory to each of these processes -- meaning at least 1 executor for both batch and streaming with 1GB of memory, and at least 1 and probably more cores. The serving layer should probably have 1GB+ of memory too and will use as many cores as the machine has.   This is designed for a cluster of machines, I suppose, but nothing prevents you from running everything on one machine. ... View more

Have you set it to start a generation based on the amount of input received? that could be triggering the new computation.   That said are you sure it only has part of the input? it's possible the zipped file sizes aren't that comparable.   Yes, you simply don't have enough memory allocated to your JVM. Your system memory doesn't matter if you haven't let the JVM use much of it. This is in local mode right? you need to use -Xmx to give more heap.   Yes it will use different tmp directories for different jobs. That's normal. ... View more

Is some of the environment setup only happening in your shell config that is triggered for interactive shells? The problem is fairly clear -- env not setup, and the question is why, but it's not really a Spark issue per se. ... View more

Right. What user is used in each case? ... View more

Whatever user you are running this as doesn't seem to have the PATH or env variables set up. See the first error:   hadoop: command not found ... View more

You're probably beyond my knowledge. But the immediate error is easy enough to understand; it can't find the Hive classes, so something is still wrong there. I see a typo in your path for example; there are two jars separated by " :=" Is it just that? ... View more

So, Spark SQL is shipped unchanged from upstream. It should mostly work as-is, as a result. It is not formally supported, as it's still an alpha component. Here in particular, have a look at other threads on this forum. I think the issue is that Spark SQL is not yet compatible with the later version of Hive in CDH, so it's not built with Hive support. Some of it should still work, but you have to add the Hive JARs to the classpath at least. ... View more

So, your app only has 3 cores from YARN? then your app can only be executing 3 tasks in parallel. I'm not sure how many receivers you are starting, but is that less?   It sounds like you expected much more resource to be avialable, so I'd go look at your YARN config and what's using the resource and compare to what Spark is actually requesting. ... View more

Go the the Spark UI and look at the top of the screen -- click Executors ... View more

You usually use --executory-memory to set executor memory but I don't think it matters.  You also generally do not use env variables to configure spark-shell. Although it might be giving the desird results, i'd use standard command line flags.   It sounds like simpler jobs are working. While you request 8 executors do you actually get them from YARN? go look at your executors tab.   ... View more

OK, what I'm interested in is how many executor slots you have. How many machines, how many executors, how many cores per executor? we want to confirm it's at least as many as the number of receivers.   what about a simpler test involving a file-based DStream? if that works then it rules out much except the custom DStream. ... View more