You would need to contact Cloudera Support if you believe it's a problem, if you have a support contract. I have successfully added Spark Gateway nodes after a cluster is live though without issues, so I suspect it's something else at work here. ... View more

It sounds like you want to have one process, not two then, if the two phases are so tied together. Also consider using a message queue like kafka and spark streaming to process the output of one separate job in another in near-real-time. I would not over-complicate it.   Tachyon is also an option but as far as I know it's not necessarily finished or completely integrate with Spark. I don't know if it will be. ... View more

An RDD is bound to an application, so it can't be shared across apps. You simply persist the data (e.g. on HDFS) and read it from the other app as an RDD.   I know people think that is slow, or slow-er than sharing an RDD somehow, but it isn't if you think about what's necessary to maintain fault tolerance across apps. You'd still be persisting something somewhere besides memory. And HDFS caching can make a lot of the reading from HDFS an in-mem operation anyway. ... View more

That I don't know. THere should be something in the logs at startup, and that should be available pretty soon. I would expect you can see the logs with that command. It could be some other issue with the ports and so on, but then I think you'd see errors from YARN that it can't get to the AM container or something. ... View more

You can background the spark-submit process like any other linux process, by putting it into the background in the shell. In your case, the spark-submit job actually then runs the driver on YARN, so, it's baby-sitting a process that's already running asynchronously on another machine via YARN. Running is good; it means all is well. You can redirect this log output where you like.   Killing the driver will cause YARN to restart it, in yarn-cluster mode. You want to kill the spark-submit process, really.   I don't know why you don't see logs. Try browing to the Spark UI of the driver to see what's happening. ... View more

I suspect it's some issue in the version of tar you may have on your system? BSD vs Gnu? Just a guess. That or maybe a corrupted file? The latest rmr2 archive uncompressed OK for me on OS X. https://github.com/RevolutionAnalytics/rmr2/releases ... View more

Hm, is that surprising? You described why it is faster in your message. The second time, "result" does not have to be recomputed since it is available on disk. It is the result of a potentially expensive shuffle operation (coalesce) ... View more