@Neeraj Sabharwal hmmm.. So this is the part where the show ends for me: Launching java with spark-submit command /usr/hdp/2.3.4.0-3485/spark/bin/spark-submit "sparkr-shell" /tmp/Rtmp69Q264/backend_portae4c24444ac20 So now I checked if spark-submit works by running the following example: cd $SPARK_HOME sudo -u spark ./bin/spark-submit --class org.apache.spark.examples.SparkPi--master yarn-client --num-executors 3--driver-memory 512m--executor-memory 512m--executor-cores 1 lib/spark-examples*.jar 10 And the result is Lots of these: INFO Client: Application report for application_1455610402042_0021 (state: ACCEPTED) Then: SparkException: Yarn application has already ended! It might have been killed or unable to launch application master. In the file, the whole error can be found spark-submit-error.txt Am I missing something in Spark setup? ... View more

@Neeraj Sabharwal I killed process, also restarted spark from ambari. If I run sudo netstat -anp | grep 9001 I dont see anything. I also have this one in my bashrc on the node where Im running sparkR: export EXISTING_SPARKR_BACKEND_PORT=9001 Funny thing, if i run sparkR with my centos user I get the error mentioned in the original post. If i run sudo -u spark sparkR then I get: Error in socketConnection(port = monitorPort) : cannot open the connection In addition: Warning message: In socketConnection(port = monitorPort) : localhost:53654 cannot be opened ... View more

@Artem Ervits Ive seen this one as well, dont see a big difference between this one and 1.5.2 I have SPARK_HOME and JAVA_HOME defined. My hive-site.xml is also on its place. If I scroll down to the SparkR part: R is installed on all the nodes. by the way, when I run sparkR, I dont get the nice Spark graphic (logo) seems as if Im starting just R. ... View more

@Neeraj Sabharwal running sudo netstat -anp | grep 9001 returns: unix 2 [ ACC ] STREAM LISTENING 9001 1202/master private/proxywrite ... View more

Im looking at the code: https://github.com/amplab-extras/SparkR-pkg/blob/master/pkg/src/src/main/scala/edu/berkeley/cs/amplab/sparkr/SparkRRunner.scala env variable EXISTING_SPARKR_BACKEND_PORT can be defined through bashrc, the try-catch that returns my error is the following: tryCatch({ connectBackend("localhost", backendPort) error = function(err) { stop("Failed to connect JVM\n") Isnt it interesting that localhost is written in it this way? Or is there an explanation for it? ... View more

What I had to do was restart ambari-server to get rid of this error. ... View more

@Artem Ervits Thank you for the link. Educative reading 🙂 The clients are installed (HDFS, YARN and Hive are, I believe, required). Ive been doing some researcher and Ive also checked the link you proposed. Adding various variables in my bashrc didnt work. I read also this document: https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/NativeLibraries.html. If I run hadoop checknative -aI get, among other things, the following: hadoop: true /usr/hdp/2.3.4.0-3485/hadoop/lib/native/libhadoop.so.1.0.0 snappy: true /usr/hdp/2.3.4.0-3485/hadoop/lib/native/libsnappy.so.1 Seems as if Hadoop sees the native libraries, but Spark does not? ... View more

More an info than an answer: proposal to join the apache incubator: http://googlecloudplatform.blogspot.ch/2016/01/Dataflow-and-open-source-proposal-to-join-the-Apache-Incubator.html acceptance: https://wiki.apache.org/incubator/BeamProposal ... View more