question sparkR - Error in socketConnection(port = monitorPort) in Support Questions

sparkR - Error in socketConnection(port = monitorPort)

marko_kole — Wed, 17 Feb 2016 00:04:00 GMT

I have centos 7.1.

On my multinode Hadoop cluster (2.3.4) I have , through Ambari, installed spark 1.5.2. I am trying to connect to sparkR from CLI and after I run sparkR I get the following error:

Error in value[[3L]](cond) : Failed to connect JVM In addition: Warning message: In socketConnection(host = hostname, port = port, server = FALSE, : localhost:9001 cannot be opened

The port (9001) is opened on the namenode (where Im running sparkR) Do you have any ideas what Im doing wrong? Ive seen this link: http://hortonworks.com/hadoop-tutorial/apache-spark-1-5-1-technical-preview-with-hdp-2-3/

and I followed also this link:

http://www.jason-french.com/blog/2013/03/11/installing-r-in-linux/

To install R on all datanodes. I appreicate your contribution.

Re: sparkR - Error in socketConnection(port = monitorPort)

aervits — Wed, 17 Feb 2016 00:05:32 GMT

@marko

please use the steps here http://hortonworks.com/hadoop-tutorial/apache-spark-1-6-technical-preview-with-hdp-2-3/

Re: sparkR - Error in socketConnection(port = monitorPort)

nsabharwal — Wed, 17 Feb 2016 00:05:54 GMT

@marko

I recommend this http://hortonworks.com/hadoop/spark/#section_6 but your link is good too based on your spark version.

Re: sparkR - Error in socketConnection(port = monitorPort)

nsabharwal — Wed, 17 Feb 2016 00:09:34 GMT

@marko Now, let's troubleshoot 9001 issue.

netstat -anp | grep 9001 --> whats the output?

Re: sparkR - Error in socketConnection(port = monitorPort)

marko_kole — Wed, 17 Feb 2016 00:13:09 GMT

Im looking at the code:

https://github.com/amplab-extras/SparkR-pkg/blob/master/pkg/src/src/main/scala/edu/berkeley/cs/amplab/sparkr/SparkRRunner.scala

env variable EXISTING_SPARKR_BACKEND_PORT can be defined through bashrc,

the try-catch that returns my error is the following:

tryCatch({    
	connectBackend("localhost", backendPort)  
	error = function(err) {    
		stop("Failed to connect JVM\n")

Isnt it interesting that localhost is written in it this way? Or is there an explanation for it?

Re: sparkR - Error in socketConnection(port = monitorPort)

marko_kole — Wed, 17 Feb 2016 00:16:35 GMT

@Neeraj Sabharwal

running sudo netstat -anp | grep 9001

returns:

unix 2 [ ACC ] STREAM LISTENING 9001 1202/master private/proxywrite

Re: sparkR - Error in socketConnection(port = monitorPort)

nsabharwal — Wed, 17 Feb 2016 00:17:31 GMT

@marko ps -ef | grep 1202

If you don't need it then kill it ...

Re: sparkR - Error in socketConnection(port = monitorPort)

aervits — Wed, 17 Feb 2016 00:17:47 GMT

@marko check whether firewall is blocking the port on each node.

Re: sparkR - Error in socketConnection(port = monitorPort)

marko_kole — Wed, 17 Feb 2016 00:20:40 GMT

@Artem Ervits

Ive seen this one as well, dont see a big difference between this one and 1.5.2

I have SPARK_HOME and JAVA_HOME defined.

My hive-site.xml is also on its place.

If I scroll down to the SparkR part: R is installed on all the nodes.

by the way, when I run sparkR, I dont get the nice Spark graphic (logo) seems as if Im starting just R.

Re: sparkR - Error in socketConnection(port = monitorPort)

marko_kole — Wed, 17 Feb 2016 00:32:48 GMT

@Neeraj Sabharwal

I killed process, also restarted spark from ambari.

If I run sudo netstat -anp | grep 9001 I dont see anything.

I also have this one in my bashrc on the node where Im running sparkR:

export EXISTING_SPARKR_BACKEND_PORT=9001

Funny thing, if i run sparkR with my centos user I get the error mentioned in the original post.

If i run sudo -u spark sparkR then I get:

Error in socketConnection(port = monitorPort) : cannot open the connection In addition: Warning message: In socketConnection(port = monitorPort) : localhost:53654 cannot be opened

Re: sparkR - Error in socketConnection(port = monitorPort)

nsabharwal — Wed, 17 Feb 2016 00:40:20 GMT

@marko Interesting...see this

http://pastebin.com/VefGqMea

Re: sparkR - Error in socketConnection(port = monitorPort)

marko_kole — Wed, 17 Feb 2016 01:03:10 GMT

@Neeraj Sabharwal

hmmm..

So this is the part where the show ends for me:

Launching java with spark-submit command /usr/hdp/2.3.4.0-3485/spark/bin/spark-submit "sparkr-shell" /tmp/Rtmp69Q264/backend_portae4c24444ac20

So now I checked if spark-submit works by running the following example:

cd $SPARK_HOME

sudo -u spark ./bin/spark-submit --class org.apache.spark.examples.SparkPi--master yarn-client --num-executors 3--driver-memory 512m--executor-memory 512m--executor-cores 1 lib/spark-examples*.jar 10

And the result is

Lots of these:

INFO Client: Application report for application_1455610402042_0021 (state: ACCEPTED)

Then:

SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.

In the file, the whole error can be found

spark-submit-error.txt

Am I missing something in Spark setup?

Re: sparkR - Error in socketConnection(port = monitorPort)

nsabharwal — Wed, 17 Feb 2016 01:15:39 GMT

@marko See this http://pastebin.com/uDEcpmga

Re: sparkR - Error in socketConnection(port = monitorPort)

marko_kole — Wed, 17 Feb 2016 01:19:55 GMT

@Neeraj Sabharwal

I ran the same spark-submit command with ONE difference:

--master was yarn-cluster

I came to the status FINISHED:

INFO Client: Application report for application_1455610402042_0022 (state: FINISHED)

and it ended up with this:

Exception in thread "main" org.apache.spark.SparkException: Application application_1455610402042_0022 finished with failed status at org.apache.spark.deploy.yarn.Client.run(Client.scala:974) at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1020) at org.apache.spark.deploy.yarn.Client.main(Client.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497)

...

16/02/16 18:15:58 INFO ShutdownHookManager: Shutdown hook called 16/02/16 18:15:58 INFO ShutdownHookManager: Deleting directory /tmp/spark-3ba9f87c-18c2-4d0d-b360-49fa10408631

Re: sparkR - Error in socketConnection(port = monitorPort)

nsabharwal — Wed, 17 Feb 2016 01:57:27 GMT

@marko yarn log -applicationid application_1455610402042_0021

Output of the above command? I hope its not failing because of memory

Re: sparkR - Error in socketConnection(port = monitorPort)

marko_kole — Wed, 17 Feb 2016 16:45:03 GMT

@Neeraj Sabharwal

Ive ran the command you recommended and Im getting an error saying

Error: Could not find or load main class log

I have setup another cluster - namenode + 3 datanodes using ambari. Ive followed this link:

http://hortonworks.com/hadoop-tutorial/apache-spark-1-6-technical-preview-with-hdp-2-3/

I installed R on all the nodes.

All examples worked until I came to the sparkR:

Launching java with spark-submit command /usr/hdp/2.3.4.0-3485/spark/bin/spark-submit "sparkr-shell" /tmp/Rtmphs2DlM/backend_port3b7b4c9a912b 16/02/17 09:37:36 WARN SparkConf: The configuration key 'spark.yarn.applicationMaster.waitTries' has been deprecated as of Spark 1.3 and and may be removed in the future. Please use the new key 'spark.yarn.am.waitTime' instead.

Error in socketConnection(port = monitorPort) :

cannot open the connection

In addition: Warning message:

In socketConnection(port = monitorPort) : localhost:40949 cannot be opened

Ive opened all the ports (1-65535) and port 0 for the namenode and the datanodes.

@Artem Ervits - do you have any idea what I am missing?

Re: sparkR - Error in socketConnection(port = monitorPort)

marko_kole — Wed, 17 Feb 2016 17:27:42 GMT

Now I installed Spark 1.6.0 just to test if Ambari makes some changes during spark installation: Same result:

Error in socketConnection(port = monitorPort) : cannot open the connection In addition: Warning message: In socketConnection(port = monitorPort) : localhost:51604 cannot be opened

Could it be YARN?

Re: sparkR - Error in socketConnection(port = monitorPort)

aervits — Wed, 17 Feb 2016 19:15:24 GMT

@marko please check whether firewall is blocking.

Re: sparkR - Error in socketConnection(port = monitorPort)

marko_kole — Wed, 17 Feb 2016 19:24:43 GMT

@Artem Ervits

I ran:

sudo systemctl status firewalld

And the result is this:

firewalld.service Loaded: not-found (Reason: No such file or directory) Active: inactive (dead)

Re: sparkR - Error in socketConnection(port = monitorPort)

aervits — Wed, 17 Feb 2016 19:27:40 GMT

Is this centos7? If not try below, also make sure to do that on all nodes

sudo service iptables stop