question Re: sparkR with HDP 2.4 deployed on AWS ec2 connection error in Archives of Support Questions (Read Only)

sparkR with HDP 2.4 deployed on AWS ec2 connection error

fisseha_berhane — Fri, 09 Sep 2016 22:44:55 GMT

I am trying to use SparkR in Rstudio in HDP2.4 deployed on AWS EC2 cluster. I installed R, Rstudio and other R packages but after I login to R and try to start spark context, I encountered the problem below.

Sys.setenv(SPARK_HOME="/usr/hdp/current/spark-client/")
.libPaths(c(file.path(Sys.getenv("SPARK_HOME"),"R","lib"),.libPaths()))

library(SparkR)

# Create a spark context and a SQL context
sc <- SparkR::sparkR.init(master = "yarn-client")

Retrying connect to server: ip-xxx-xx-xx-xx.ec2.internal/xxx.xx.xx.xx:8050. Already tried 49 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
16/09/09 11:41:32

Re: sparkR with HDP 2.4 deployed on AWS ec2 connection error

cstanca — Sat, 10 Sep 2016 03:48:22 GMT

@Fish Berh

I assume that your R-Studio is on your laptop. It seems that you are trying to access an internal IP from your laptop. You need to reference a public IP or the public URI for your server.

Re: sparkR with HDP 2.4 deployed on AWS ec2 connection error

cstanca — Sat, 10 Sep 2016 03:53:56 GMT

@Fish Berh

You may also want to check a few options here:

https://blog.rstudio.org/tag/sparkr/

Re: sparkR with HDP 2.4 deployed on AWS ec2 connection error

fisseha_berhane — Sat, 10 Sep 2016 03:57:41 GMT

Actually, I logged in to Rstudio using the address of the server:

ec2-yy-yyy-yyy-yy.compute-1.amazonaws.com:8787

Re: sparkR with HDP 2.4 deployed on AWS ec2 connection error

cstanca — Sat, 10 Sep 2016 06:27:30 GMT

@Fish Berh

Could you vote and accept my response? I suggested using the public URI of the server.

Re: sparkR with HDP 2.4 deployed on AWS ec2 connection error

fisseha_berhane — Mon, 12 Sep 2016 21:30:06 GMT

I used public URI of the server to login and got the error.