question Re: Error in library(SparkR) when running R with Spark on Sandbox 2.3.2 in Archives of Support Questions (Read Only)

Error in library(SparkR) when running R with Spark on Sandbox 2.3.2

George23504 — Thu, 24 Dec 2015 05:43:53 GMT

Working with Spark 1.4.1 on Sandbox 2.3.2. python worked fine with Spark. Scala OK after installing scala 2.11.7. Cannot get R to work with Spark. Installed R 3.2.2 through yum. Then ./bin/sparkR was able to start R shell, but without the "Welcome to SparkR!" message. When running ./bin/spark-submit examples/src/main/r/dataframe.R, got an "Error in library(SparkR): there is no package called 'SparkR'".

Re: Error in library(SparkR) when running R with Spark on Sandbox 2.3.2

nsabharwal — Thu, 24 Dec 2015 05:49:42 GMT

@Chungchu Hsieh

example https://gist.github.com/shivaram/d0cd4aa5c4381edd6...

http://stackoverflow.com/questions/29769357/error-...

Try this

Installed:

R-core.x86_64 0:3.2.2-2.el6 R-core-devel.x86_64 0:3.2.2-2.el6

Dependency Installed:

blas.x86_64 0:3.2.1-4.el6 blas-devel.x86_64 0:3.2.1-4.el6 bzip2-devel.x86_64 0:1.0.5-7.el6_0 desktop-file-utils.x86_64 0:0.15-9.el6

fontconfig-devel.x86_64 0:2.8.0-5.el6 freetype-devel.x86_64 0:2.3.11-15.el6_6.1 gcc-gfortran.x86_64 0:4.4.7-16.el6 kpathsea.x86_64 0:2007-60.el6_7

lapack.x86_64 0:3.2.1-4.el6 lapack-devel.x86_64 0:3.2.1-4.el6 libX11-devel.x86_64 0:1.6.0-6.el6 libXau-devel.x86_64 0:1.0.6-4.el6

libXft-devel.x86_64 0:2.3.1-2.el6 libXmu.x86_64 0:1.1.1-2.el6 libXrender-devel.x86_64 0:0.9.8-2.1.el6 libgfortran.x86_64 0:4.4.7-16.el6

libicu-devel.x86_64 0:4.2.1-12.el6 libxcb-devel.x86_64 0:1.9.1-3.el6 netpbm.x86_64 0:10.47.05-11.el6 netpbm-progs.x86_64 0:10.47.05-11.el6

pcre-devel.x86_64 0:7.8-7.el6 psutils.x86_64 0:1.17-34.el6 tcl.x86_64 1:8.5.7-6.el6 tcl-devel.x86_64 1:8.5.7-6.el6

tex-preview.noarch 0:11.85-10.el6 texinfo.x86_64 0:4.13a-8.el6 texinfo-tex.x86_64 0:4.13a-8.el6 texlive.x86_64 0:2007-60.el6_7

texlive-dvips.x86_64 0:2007-60.el6_7 texlive-latex.x86_64 0:2007-60.el6_7 texlive-texmf.noarch 0:2007-39.el6_7 texlive-texmf-dvips.noarch 0:2007-39.el6_7

texlive-texmf-errata.noarch 0:2007-7.1.el6 texlive-texmf-errata-dvips.noarch 0:2007-7.1.el6 texlive-texmf-errata-fonts.noarch 0:2007-7.1.el6 texlive-texmf-errata-latex.noarch 0:2007-7.1.el6

texlive-texmf-fonts.noarch 0:2007-39.el6_7 texlive-texmf-latex.noarch 0:2007-39.el6_7 texlive-utils.x86_64 0:2007-60.el6_7 tk.x86_64 1:8.5.7-5.el6

tk-devel.x86_64 1:8.5.7-5.el6 xdg-utils.noarch 0:1.0.2-17.20091016cvs.el6 xorg-x11-proto-devel.noarch 0:7.7-9.el6 xz-devel.x86_64 0:4.999.9-0.5.beta.20091007git.el6

Complete!

[root@sandbox bin]# ./sparkR

R version 3.2.2 (2015-08-14) -- "Fire Safety"

Platform: x86_64-redhat-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.

You are welcome to redistribute it under certain conditions.

Type 'license()' or 'licence()' for distribution details.

Natural language support but running in an English locale

R is a collaborative project with many contributors.

Type 'contributors()' for more information and

'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or

'help.start()' for an HTML browser interface to help.

Type 'q()' to quit R.

Re: Error in library(SparkR) when running R with Spark on Sandbox 2.3.2

George23504 — Thu, 24 Dec 2015 06:43:34 GMT

Thanks Neeraj. The R-core and R-core-devel packages are already installed. I also get the same messages from ./sparkR command as you've listed. However, I don't get the "Welcome to SparkR!' message which I thought should be displayed if all goes well. Furthermore, when I try

>sc <- sparkR.init()

I would get an error message - Could not find function: "sparkR.init". It appears I can run R but not R with Spark.

Re: Error in library(SparkR) when running R with Spark on Sandbox 2.3.2

nsabharwal — Mon, 19 Aug 2019 12:26:07 GMT

http://pastebin.com/QikSKsk0

Worked for Spark1.5.2

Re: Error in library(SparkR) when running R with Spark on Sandbox 2.3.2

sascha_kerbler — Mon, 28 Dec 2015 19:01:02 GMT

I had the same problem, this means that spark can not find the R directory. The Hortonworks Spark version comes without the R directory:

drwxr-xr-x 2 root root 4096 Dec 9 13:33 lib

drwxr-xr-x 6 root root 4096 Dec 3 21:30 python

drwxr-xr-x 3 root root 4096 Jul 8 22:58 R

My solution was to download the Spark standalone version and copy the R directory to my Hortonworks Spark client.

Re: Error in library(SparkR) when running R with Spark on Sandbox 2.3.2

George23504 — Tue, 29 Dec 2015 07:45:59 GMT

Thanks Sascha. That's what I observed as well. Your solution seems a reasonable workaround.