@Neeraj Sabharwal I have now installed 1.5.2 on one of the datanodes, I can run spark-shell, when I run sparkR I get this issue: Launching java with spark-submit command /usr/hdp/2.3.4.0-3485/spark/bin/spark-submit "sparkr-shell" /tmp/Rtmp6iFIT6/backend_port15af7623f5dd 16/02/17 14:35:18 WARN internal.ThreadLocalRandom: Failed to generate a seed from SecureRandom within 3 seconds. Not enough entrophy? Error in socketConnection(port = monitorPort) : cannot open the connection In addition: Warning message: In socketConnection(port = monitorPort) : localhost:44867 cannot be opened Pretty much the same problem. ... View more

@Neeraj Sabharwal Ive manually installed 1.6.0 on the same node where Im running spark 1.5.2. Now I ll try doing it on one datanode - I ll install 1.5.2 manually. By the way, Neeraj, have you seen the CentOS 7.2 dialog Im having with Artem. Let me know if you have a take on that one. ... View more

@Artem Ervits, @Neeraj Sabharwal Well Here is my version: CentOS Linux release 7.2.1511 (Core) The cloud services I use have, in their menu, CentOS 7.1, I think they upgraded the systems, left 7.1 on the menu but the instances are 7.2 Now im a bit upset. Do you think this could really be it? That 7.2 version is the problem? ... View more

@Neeraj Sabharwal Nothing nice about this issue, its a show stopper if you work for researchers. 🙂 What do you mean by different node? And run it how? ... View more

@Neeraj Sabharwal No, the spark installation was done through Ambari. I installed, accepted all the defaults and ran the test examples from the TP website.Funny thing, when installing, you get to choose 1.4.2 version of spark, while 1.5.2 is installed. ... View more

@Artem ErvitsI tried that as well. Here the status command: sudo service iptables status And the result: Redirecting to /bin/systemctl status iptables.service ● iptables.service Loaded: not-found (Reason: No such file or directory) Active: inactive (dead) I tried running the iptables and firewalld commands on the datanodes and they all seem inactive (same results as above). Centos 7.2. ... View more

@Artem Ervits I ran: sudo systemctl status firewalld And the result is this: firewalld.service Loaded: not-found (Reason: No such file or directory) Active: inactive (dead) ... View more

Now I installed Spark 1.6.0 just to test if Ambari makes some changes during spark installation: Same result: Error in socketConnection(port = monitorPort) : cannot open the connection In addition: Warning message: In socketConnection(port = monitorPort) : localhost:51604 cannot be opened Could it be YARN? ... View more

@Neeraj Sabharwal Ive ran the command you recommended and Im getting an error saying Error: Could not find or load main class log I have setup another cluster - namenode + 3 datanodes using ambari. Ive followed this link: http://hortonworks.com/hadoop-tutorial/apache-spark-1-6-technical-preview-with-hdp-2-3/ I installed R on all the nodes. All examples worked until I came to the sparkR: Launching java with spark-submit command /usr/hdp/2.3.4.0-3485/spark/bin/spark-submit "sparkr-shell" /tmp/Rtmphs2DlM/backend_port3b7b4c9a912b 16/02/17 09:37:36 WARN SparkConf: The configuration key 'spark.yarn.applicationMaster.waitTries' has been deprecated as of Spark 1.3 and and may be removed in the future. Please use the new key 'spark.yarn.am.waitTime' instead. Error in socketConnection(port = monitorPort) : cannot open the connection In addition: Warning message: In socketConnection(port = monitorPort) : localhost:40949 cannot be opened > Ive opened all the ports (1-65535) and port 0 for the namenode and the datanodes. @Artem Ervits - do you have any idea what I am missing? ... View more

@Neeraj Sabharwal I ran the same spark-submit command with ONE difference: --master was yarn-cluster I came to the status FINISHED: INFO Client: Application report for application_1455610402042_0022 (state: FINISHED) and it ended up with this: Exception in thread "main" org.apache.spark.SparkException: Application application_1455610402042_0022 finished with failed status at org.apache.spark.deploy.yarn.Client.run(Client.scala:974) at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1020) at org.apache.spark.deploy.yarn.Client.main(Client.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) ... 16/02/16 18:15:58 INFO ShutdownHookManager: Shutdown hook called 16/02/16 18:15:58 INFO ShutdownHookManager: Deleting directory /tmp/spark-3ba9f87c-18c2-4d0d-b360-49fa10408631 ... View more