Running R program on HDP
In general, you have following options when running R on Hortonworks Data Platform (HDP) -
o RHadoop (rmr) - R program written in MapReduce paradigm. MapReduce is not a vendor specific API and any program written with MapReduce is portable across Hadoop distributions.
https://github.com/RevolutionAnalytics/RHadoop/wiki/rmr
o Hadoop Streaming - R program written to make use of Hadoop Streaming but the program structure still aligns with MapReduce. Above benefit still applies.
o RJDBC - This example does not require the R programs to be written using MapReduce and still remains 100% native R APIs without any third party packages.
Here is a tutorial with a video, sample data and R script:
http://hortonworks.com/hadoop-tutorial/using-revolution-r-enterprise-tutorial-hortonworks-sandbox/
Using RJDBC, the R program can have Hadoop parallelize pre-processing and filtering. R submits a query to Hive or SparkSQL making use of distributed and parallel processing. Then uses existing R models, as is & without any changes or use of any proprietary APIs. Typically speaking, any data science application involves a ton of prepping which is usually 75% of the work. RJDBC allows pushing that work to Hive to take advantage of distributed computing.
o Spark R - Lastly, the Spark R interface which is a newer component in Spark. SparkR is an R package that provides a light-weight frontend to use Apache Spark from R. This component is available since Spark 1.4.1 (current version 1.5.2)
Here are some details on it -
https://spark.apache.org/docs/latest/sparkr.html
And the available API -
