Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Unable to Create Spark Context


Unable to Create Spark Context

New Contributor



I am using CDH 5.12.2 on a 4 node cluster. I amd trying to run a talend (talend studio 6.5.1) job, currently i am running into a issue which is throwing an exception.  please find the exception below.  


This job mainly transfers csv files from an sftp server to hadoop cluster using Talend jobs.


what i observed is once the job starts, Talend is creating a .staging directory under the xyzuser home directory in HDFS and transfereing some jar files and while doing this at some point the below exception is thrown and job is failing.


I tried a similar data ingestion MapReduce job from Talend which is working successfully. 


PriviledgedActionException as:xyzuser (auth:PROXY) via mapred (auth:SIMPLE) File does not exist: /user/xyzuser/.staging/job_1524652846082_0089/job_1524652846082_0089.summary
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(
	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$
	at org.apache.hadoop.ipc.RPC$
	at org.apache.hadoop.ipc.Server$Handler$
	at org.apache.hadoop.ipc.Server$Handler$
	at Method)
	at org.apache.hadoop.ipc.Server$

 I really appreciate if someone help me understand what the issue is?

Don't have an account?
Coming from Hortonworks? Activate your account here