[ec2-user@ip-10-16-129-152 bin]$ ./sqlline.py Setting property: [incremental, false] Setting property: [isolation, TRANSACTION_READ_COMMITTED] issuing: !connect jdbc:phoenix:localhost:2181:/hbase none none org.apache.phoenix.jdbc.PhoenixDriver Connecting to jdbc:phoenix:localhost:2181:/hbase SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/lib/phoenix/phoenix-4.13.0-HBase-1.4-client.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 18/05/14 04:07:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 18/05/14 04:17:47 WARN ipc.CoprocessorRpcChannel: Call failed on IOException org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Mon May 14 04:07:47 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:07:47 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: /<IP ADDRESS>:16020 Mon May 14 04:07:47 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: /<IP ADDRESS>:16020 Mon May 14 04:07:47 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: /<IP ADDRESS>:16020 Mon May 14 04:07:48 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: /<IP ADDRESS>:16020 Mon May 14 04:07:50 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:07:55 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:08:02 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:08:15 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:08:28 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:08:41 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:08:54 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:09:18 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:09:41 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:10:04 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:10:27 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:10:50 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:11:13 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:11:37 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:12:00 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:12:23 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:12:46 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:13:09 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:13:32 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:13:56 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:14:19 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:14:42 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:15:05 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:15:28 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:15:51 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:16:15 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:16:38 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:17:01 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:17:24 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:17:47 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:164) at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:101) at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callMethod(CoprocessorRpcChannel.java:56) at org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService$Stub.getVersion(MetaDataProtos.java:16649) at org.apache.phoenix.query.ConnectionQueryServicesImpl$5.call(ConnectionQueryServicesImpl.java:1198) at org.apache.phoenix.query.ConnectionQueryServicesImpl$5.call(ConnectionQueryServicesImpl.java:1190) at org.apache.hadoop.hbase.client.HTable$15.call(HTable.java:1775) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:389) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:94) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:409) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:405) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:422) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:327) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:94) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.execService(ClientProtos.java:37083) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1834) at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:97) at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:88) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:142) ... 10 more Caused by: java.net.NoRouteToHostException: No route to host at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.hbase.ipc.BlockingRpcConnection.setupConnection(BlockingRpcConnection.java:256) at org.apache.hadoop.hbase.ipc.BlockingRpcConnection.setupIOstreams(BlockingRpcConnection.java:437) at org.apache.hadoop.hbase.ipc.BlockingRpcConnection.writeRequest(BlockingRpcConnection.java:540) at org.apache.hadoop.hbase.ipc.BlockingRpcConnection.tracedWriteRequest(BlockingRpcConnection.java:520) at org.apache.hadoop.hbase.ipc.BlockingRpcConnection.access$200(BlockingRpcConnection.java:85) at org.apache.hadoop.hbase.ipc.BlockingRpcConnection$4.run(BlockingRpcConnection.java:724) at org.apache.hadoop.hbase.ipc.HBaseRpcControllerImpl.notifyOnCancel(HBaseRpcControllerImpl.java:240) at org.apache.hadoop.hbase.ipc.BlockingRpcConnection.sendRequest(BlockingRpcConnection.java:699) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:420) ... 18 more 18/05/14 04:17:47 WARN client.HTable: Error calling coprocessor service org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService for row java.util.concurrent.ExecutionException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Mon May 14 04:07:47 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: java.net.NoRouteToHostException: No route to host Mon May 14 04:07:47 UTC 2018, RpcRetryingCaller{globalStartTime=1526270864184, pause=100, retries=35}, java.io.IOException: Call to /<IP ADDRESS>:16020 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: /<IP ADDRESS>:16020 ... View more

I am trying to run an application in yarn cluster mode. Here are setting of the shell script: spark-submit --class "com.Myclass" \ --num-executors 2 \ --executor-cores 2 \ --master yarn \ --supervise \ --deploy-mode cluster \ ../target/ \ My application is working fine in yarn-client and local mode. EXCERPT FROM SPARK-SUBMIT UNDER YARN CLUSTER MODE &&&&&&&&&&&&&&&&&&&&&& HADOOP HOME /usr/lib/hadoop &&&&&&&&&&&&&&&&&&&&&& HADOOP_CONF_DIR /usr/lib/hadoop/etc/hadoop ... Diagnostics: Exception from container-launch. Container id: container_1454984479786_0006_02_000001 Exit code: 15 Stack trace: ExitCodeException exitCode=15: at org.apache.hadoop.util.Shell.runCommand(Shell.java:543) at org.apache.hadoop.util.Shell.run(Shell.java:460) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:720) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:210) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:262) Further I am getting following error ERROR DETAILS FROM YARN LOGS APPLICATIONID INFO : org.apache.spark.deploy.yarn.ApplicationMaster - Registered signal handlers for [TERM, HUP, INT] DEBUG: org.apache.hadoop.util.Shell - Failed to detect a valid hadoop home directory java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set. at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:307) at org.apache.hadoop.util.Shell.<clinit>(Shell.java:332) at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:79) at org.apache.hadoop.yarn.conf.YarnConfiguration.<clinit>(YarnConfiguration.java:590) at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.newConfiguration(YarnSparkHadoopUtil.scala:62) at org.apache.spark.deploy.SparkHadoopUtil.<init>(SparkHadoopUtil.scala:52) at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.<init>(YarnSparkHadoopUtil.scala:47) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at java.lang.Class.newInstance(Class.java:374) at org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:386) at org.apache.spark.deploy.SparkHadoopUtil$.yarn$lzycompute(SparkHadoopUtil.scala:384) at org.apache.spark.deploy.SparkHadoopUtil$.yarn(SparkHadoopUtil.scala:384) at org.apache.spark.deploy.SparkHadoopUtil$.get(SparkHadoopUtil.scala:401) at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:623) at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala) I tried modifying spark-env.sh like following and I see Hadoop_Home logged but still getting error: Modified added following entries to spark-env.sh export HADOOP_HOME="/usr/lib/hadoop" echo "&&&&&&&&&&&&&&&&&&&&&& HADOOP HOME " echo "$HADOOP_HOME" export HADOOP_CONF_DIR="$HADOOP_HOME/etc/hadoop" echo "&&&&&&&&&&&&&&&&&&&&&& HADOOP_CONF_DIR " echo "$HADOOP_CONF_DIR" ... View more