New Contributor
Posts: 1
Registered: ‎05-16-2018

Unable to Create Spark Context

[ Edited ]



I am using CDH 5.12.2 on a 4 node cluster. I amd trying to run a talend (talend studio 6.5.1) job, currently i am running into a issue which is throwing an exception.  please find the exception below.  


This job mainly transfers csv files from an sftp server to hadoop cluster using Talend jobs.


what i observed is once the job starts, Talend is creating a .staging directory under the xyzuser home directory in HDFS and transfereing some jar files and while doing this at some point the below exception is thrown and job is failing.


I tried a similar data ingestion MapReduce job from Talend which is working successfully. 


PriviledgedActionException as:xyzuser (auth:PROXY) via mapred (auth:SIMPLE) File does not exist: /user/xyzuser/.staging/job_1524652846082_0089/job_1524652846082_0089.summary
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(
	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$
	at org.apache.hadoop.ipc.RPC$
	at org.apache.hadoop.ipc.Server$Handler$
	at org.apache.hadoop.ipc.Server$Handler$
	at Method)
	at org.apache.hadoop.ipc.Server$

 I really appreciate if someone help me understand what the issue is?