Solved this by having following property defined in workflow.xml. <configuration> <property> <name>oozie.launcher.mapreduce.job.user.classpath.first</name> <value>true</value> </property> ..... </configuration>
... View more
Have a look at the parent packaging pom to see some additional settings like this that affect the CDH packaging. I don't know how much they're documented beyond this as it's generally rare for anyone to try to rebuild the source. Still, it ought not be hard.
... View more
Your signature is just a little bit off. The result of a join is not a triple, but a tuple whose second element is a tuple. You have: (_, (_, _),(_,_,device)) but I think you need: (_, ((_, _),(_,_,device)))
... View more
It will make a difference insofar as the driver program will run either out on the cluster (yarn-cluster) or locally (yarn-client). The same issue remains -- the processes need to talk to each other on certain ports. But it affects where the driver is and that affects what machine's ports need to be open. For example, if your ports are all open within your cluster, I expect that yarn-cluster works directly.
... View more
I believe it was added in 1.1, yes. I don't have a streaming app driver handy, so maybe double-check -- you will see an obvious Streaming tab if it's there. Without guaranteeing anything, I think the next CDH will have 1.1, and at any time you can run your own Spark jobs with any version under YARN.
... View more
I think you imported just about everything except the one thing you need to get implicit conversions that unlock the functions in PairRDDFunctions, which is where join() is defined. You need: import org.apache.spark.SparkContext._ In the shell this is imported by default.
... View more
The default is that you manually trigger model builds. But you can configure it to build after a certain amount of time has elapsed, or a certain number of data points have been written. See model.time-threshold and model.data-threshold. Yes, all data points cause in-memory model updates no matter how they arrive.
... View more
