Error in library(SparkR) when running R with Spark on Sandbox 2.3.2

[#1]

Working with Spark 1.4.1 on Sandbox 2.3.2. python worked fine with Spark. Scala OK after installing scala 2.11.7. Cannot get R to work with Spark. Installed R 3.2.2 through yum. Then ./bin/sparkR was able to start R shell, but without the "Welcome to SparkR!" message. When running ./bin/spark-submit examples/src/main/r/dataframe.R, got an "Error in library(SparkR): there is no package called 'SparkR'".

[#2]

@Chungchu Hsieh

example https://gist.github.com/shivaram/d0cd4aa5c4381edd6...

http://stackoverflow.com/questions/29769357/error-...

Try this

Installed:

R-core.x86_64 0:3.2.2-2.el6 R-core-devel.x86_64 0:3.2.2-2.el6

Dependency Installed:

blas.x86_64 0:3.2.1-4.el6 blas-devel.x86_64 0:3.2.1-4.el6 bzip2-devel.x86_64 0:1.0.5-7.el6_0 desktop-file-utils.x86_64 0:0.15-9.el6

fontconfig-devel.x86_64 0:2.8.0-5.el6 freetype-devel.x86_64 0:2.3.11-15.el6_6.1 gcc-gfortran.x86_64 0:4.4.7-16.el6 kpathsea.x86_64 0:2007-60.el6_7

lapack.x86_64 0:3.2.1-4.el6 lapack-devel.x86_64 0:3.2.1-4.el6 libX11-devel.x86_64 0:1.6.0-6.el6 libXau-devel.x86_64 0:1.0.6-4.el6

libXft-devel.x86_64 0:2.3.1-2.el6 libXmu.x86_64 0:1.1.1-2.el6 libXrender-devel.x86_64 0:0.9.8-2.1.el6 libgfortran.x86_64 0:4.4.7-16.el6

libicu-devel.x86_64 0:4.2.1-12.el6 libxcb-devel.x86_64 0:1.9.1-3.el6 netpbm.x86_64 0:10.47.05-11.el6 netpbm-progs.x86_64 0:10.47.05-11.el6

pcre-devel.x86_64 0:7.8-7.el6 psutils.x86_64 0:1.17-34.el6 tcl.x86_64 1:8.5.7-6.el6 tcl-devel.x86_64 1:8.5.7-6.el6

tex-preview.noarch 0:11.85-10.el6 texinfo.x86_64 0:4.13a-8.el6 texinfo-tex.x86_64 0:4.13a-8.el6 texlive.x86_64 0:2007-60.el6_7

texlive-dvips.x86_64 0:2007-60.el6_7 texlive-latex.x86_64 0:2007-60.el6_7 texlive-texmf.noarch 0:2007-39.el6_7 texlive-texmf-dvips.noarch 0:2007-39.el6_7

texlive-texmf-errata.noarch 0:2007-7.1.el6 texlive-texmf-errata-dvips.noarch 0:2007-7.1.el6 texlive-texmf-errata-fonts.noarch 0:2007-7.1.el6 texlive-texmf-errata-latex.noarch 0:2007-7.1.el6

texlive-texmf-fonts.noarch 0:2007-39.el6_7 texlive-texmf-latex.noarch 0:2007-39.el6_7 texlive-utils.x86_64 0:2007-60.el6_7 tk.x86_64 1:8.5.7-5.el6

tk-devel.x86_64 1:8.5.7-5.el6 xdg-utils.noarch 0:1.0.2-17.20091016cvs.el6 xorg-x11-proto-devel.noarch 0:7.7-9.el6 xz-devel.x86_64 0:4.999.9-0.5.beta.20091007git.el6

Complete!

[root@sandbox bin]# ./sparkR

R version 3.2.2 (2015-08-14) -- "Fire Safety"

Copyright (C) 2015 The R Foundation for Statistical Computing

Platform: x86_64-redhat-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.

You are welcome to redistribute it under certain conditions.

Type 'license()' or 'licence()' for distribution details.

Natural language support but running in an English locale

R is a collaborative project with many contributors.

Type 'contributors()' for more information and

'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or

'help.start()' for an HTML browser interface to help.

Type 'q()' to quit R.

>

[#3]

Thanks Neeraj. The R-core and R-core-devel packages are already installed. I also get the same messages from ./sparkR command as you've listed. However, I don't get the "Welcome to SparkR!' message which I thought should be displayed if all goes well. Furthermore, when I try

>sc <- sparkR.init()

I would get an error message - Could not find function: "sparkR.init". It appears I can run R but not R with Spark.

[#4]

http://pastebin.com/QikSKsk0

Worked for Spark1.5.2

[#5 · Accepted solution]

Hi

I had the same problem, this means that spark can not find the R directory. The Hortonworks Spark version comes without the R directory:

drwxr-xr-x 2 root root 4096 Dec 9 13:33 lib

drwxr-xr-x 6 root root 4096 Dec 3 21:30 python

drwxr-xr-x 3 root root 4096 Jul 8 22:58 R

My solution was to download the Spark standalone version and copy the R directory to my Hortonworks Spark client.

[#6]

Thanks Sascha. That's what I observed as well. Your solution seems a reasonable workaround.