Issue with setting up spark clients without ambari

[#1]

I am trying to evaluate sparklyr in a test machine with RStudio Server. Since the machine is outside the HDP cluster I installed the hadoop and spark clients and copied the config files our test hdp cluster in /etc/hadoop/conf . I set the HADOOP_CONF_DIR and YARN_CONF_DIR and SPARK_HOME to point to hdp files. Our hadoop cluster is integrated with Kerberos. I am able to run spark-shell on local mode and read hdfs files from the test cluster. I am not able to run spark-shell on yarn-client mode .

I am getting the following error in application log.

16/10/05 11:30:57 INFO yarn.ApplicationMaster: Waiting for Spark driver to be reachable.
16/10/05 11:32:00 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10.100.99.100:42948, retrying ...
16/10/05 11:33:03 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10.100.99.100:42948, retrying ...
16/10/05 11:33:03 ERROR yarn.ApplicationMaster: Uncaught exception: 
org.apache.spark.SparkException: Failed to connect to driver!

It is submitting the job and the job goto ACCEPTED state but not to RUNNING state.

6/10/05 10:43:24 INFO impl.YarnClientImpl: Submitted application application_1474880908029_0858

16/10/05 10:43:24 INFO cluster.SchedulerExtensionServices: Starting Yarn extension services with app application_1474880908029_0858 and attemptId None

16/10/05 10:43:25 INFO yarn.Client: Application report for application_1474880908029_0858 (state: ACCEPTED)

16/10/05 10:43:25 INFO yarn.Client:

client token: Token { kind: YARN_CLIENT_TOKEN, service: }

diagnostics: N/A

ApplicationMaster host: N/A

ApplicationMaster RPC port: -1

queue: default

start time: 1475660604154

final status: UNDEFINED

tracking URL: http://hostname:8088/proxy/application_1474880908029_0858/

user: dee

16/10/05 10:43:26 INFO yarn.Client: Application report for application_1474880908029_0858 (state: ACCEPTED)

16/10/05 10:43:27 INFO yarn.Client: Application report for application_1474880908029_0858 (state: ACCEPTED)

16/10/05 10:43:28 INFO yarn.Client: Application report for application_1474880908029_0858 (state: ACCEPTED)

16/10/05 10:43:29 INFO yarn.Client: Application report for application_1474880908029_0858 (state: ACCEPTED)

Here is the application log.

16/10/05 11:30:57 INFO spark.SecurityManager: Changing view acls to: deesub
16/10/05 11:30:57 INFO spark.SecurityManager: Changing modify acls to: deesub
16/10/05 11:30:57 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(deesub); users with modify permissions: Set(deesub)
16/10/05 11:30:57 INFO yarn.ApplicationMaster: Waiting for Spark driver to be reachable.
16/10/05 11:32:00 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10.100.99.100:42948, retrying ...
16/10/05 11:33:03 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10.100.99.100:42948, retrying ...
16/10/05 11:33:03 ERROR yarn.ApplicationMaster: Uncaught exception: 
org.apache.spark.SparkException: Failed to connect to driver!
	at org.apache.spark.deploy.yarn.ApplicationMaster.waitForSparkDriver(ApplicationMaster.scala:501)
	at org.apache.spark.deploy.yarn.ApplicationMaster.runExecutorLauncher(ApplicationMaster.scala:362)
	at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:204)
	at org.apache.spark.deploy.yarn.ApplicationMaster$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:672)
	at org.apache.spark.deploy.SparkHadoopUtil$anon$1.run(SparkHadoopUtil.scala:69)
	at org.apache.spark.deploy.SparkHadoopUtil$anon$1.run(SparkHadoopUtil.scala:68)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
	at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:68)
	at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:670)
	at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:697)
	at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala)
16/10/05 11:33:03 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 10, (reason: Uncaught exception: org.apache.spark.SparkException: Failed to connect to driver!)
16/10/05 11:33:03 INFO util.ShutdownHookManager: Shutdown hook called

[#2 · Accepted solution]

Did you copy /etc/spark/conf/ ? Also, create spark user and copy its headless keytab. You can find all details here.

[#3]

I missed copying the spark conf. But spark user was created with yum install of spark clients. I also copied the keytab. But still the same error.

16/10/05 20:05:55 ERROR ApplicationMaster: Failed to connect to driver at 10.100.100.110:33656, retrying ...
16/10/05 20:06:58 ERROR ApplicationMaster: Failed to connect to driver at 10.100.100.110:33656, retrying ...
16/10/05 20:06:58 ERROR ApplicationMaster: Uncaught exception: 
org.apache.spark.SparkException: Failed to connect to driver!

[#4]

You may have some network issues, if I understand the log correctly it is saying that the assigned Application master for your spark job cannot access the driver running on your external node. Can cluster nodes access your external node on ephemeral port numbers like 33656?

[#5]

Good point. It looks like it is a firewall issue.