You asked why Data Science teams claim that they cannot do most of their work on the cluster with R. My point is that it is due to the fact that R is mainly a client side studio, not that different from Eclipse (but with more tools). The article I suggested points out the hoops you have to jump through to run R across a cluster. Spark R does not really address this just yet since Spark R is simply an R interpreter that turns the instructions sets into RDDs and then executes them as Spark on the cluster. Spark R does not actually use any of the R packages to exectue logic. Take a look at the Spark R page (https://spark.apache.org/docs/1.6.0/sparkr.html) it mainly talks about creating data frames using R syntax. The section on machine learning covers Gaussian and Binomial GLM, thats it, that is Spark R at this point. If the requirements of your project can be satisfied with using these techniques then great, you can now do your work on the cluster. If not, you will need to learn Spark and Scala. Until Spark has all of the functions and algorithms that R is capable of, Spark R will not completely solve the problem. That is why data scientist that do not have a strong dev background continue to sample data to make it fit on their workstation, so that they can continue to use all of the packages that R provides. ... View more

Not generally. You can use a framework like rmr or foreach to simulate parallelism but it there is a bunch of extra work required. Most data scientists just run R on their workstation and sample data to fit within the resource constraints of their system. Once Spark catches up in terms of available algorithms I think more and more data science will be done on the cluster. Check out this blog about the different modes of R http://blog.revolutionanalytics.com/2015/06/using-hadoop-with-r-it-depends.html ... View more

Yes, and that is how I am applying my models to the demos that I have built so far. I was interested in whether it is possible to create a Spark context in a Storm Bolt. Sounds like the answer might be no. Is it? ... View more

Need to see the query statement and your current Hive/Tez/Yarm settings in order find the issue. It look like one of your containers is running out of memory but 4GB is a decent size. You may need to address your query structure or you may simply need more reducers or to increase the container maximum heap size that Tez can request. You will need to check your Yarn container sizing as well since Tez cannot ask for a larger container than Yarn will allow... ... View more

Just put your JDBC driver in the classpath and then write the connection to the DB just liek you would from any Java program. Storm is not dependent on HDFS. In fact, you don't need a Hadoop cluster to run Storm. You can read and write on every event that comes through Storm into the DB2 database.... ... View more