Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark Submit Job Does Not Copy Job Files To Azure HortonWorks Sandbox

Highlighted

Spark Submit Job Does Not Copy Job Files To Azure HortonWorks Sandbox

New Contributor

Hi,

We are trying to execute a spark submit job from our local machine to run against our Azure HortonWorks Sandbox Hadoop Environment. We are experiencing a problem where the transfer of the job files to be submitted does not copy across to the HortonWorks datanode for execution which then results in the error below. Has anyone come across this or a similar issue before? Let me know if there is more information required to help with this query.

The command we are running is: spark-submit --master yarn spark-test-0.0.3-SNAPSHOT.jar

This connects to the Azure HortonWorks environment fine (yarn configs and relevant ports appear setup/forwarded ok) but fails at the point of uploading resources, the following error is thrown:

Exception in thread "main" org.apache.hadoop.ipc.RemoteException(java.io.IOExcep tion): File /user/hdfs spark-shell/.sparkStaging/application_1495640151047_0030/ __spark_libs__9041480449031548915.zip could only be replicated to 0 nodes instea d of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are exc luded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarg et4NewBlock(BlockManager.java:1641) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTarget s(FSNamesystem.java:3198) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBloc k(FSNamesystem.java:3122) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(Nam eNodeRpcServer.java:843) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTra nslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:500) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$Cl ientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.cal l(ProtobufRpcEngine.java:640) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma tion.java:1724) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307) at org.apache.hadoop.ipc.Client.call(Client.java:1475) at org.apache.hadoop.ipc.Client.call(Client.java:1412) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEng ine.java:229) at com.sun.proxy.$Proxy12.addBlock(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB. addBlock(ClientNamenodeProtocolTranslatorPB.java:418) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryI nvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocat ionHandler.java:102) at com.sun.proxy.$Proxy13.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBl ock(DFSOutputStream.java:1455) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputSt ream(DFSOutputStream.java:1251) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStre am.java:448) 17/05/25 17:28:44 INFO util.ShutdownHookManager: Shutdown hook called 17/05/25 17:28:44 INFO util.ShutdownHookManager: Deleting directory C:\Temp\spar k-c1366f57-54eb-4bb6-ae11-26acf4b5b2f8

Don't have an account?
Coming from Hortonworks? Activate your account here