Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

error while calling spark job using yarn rest api

Highlighted

error while calling spark job using yarn rest api

Explorer

Hi All,

 

I'm able to submit spark job using both yarn-client and yarn-cluster mode successfully on a kerberized cluster from an edge/client node managed by cloudera manager using both spark on yarn and spark2.

 

 

-sh-4.1$ spark-submit --class org.apache.spark.examples.SparkPi --deploy-mode client --master yarn /opt/cloudera/parcels/CDH/lib/spark/lib/spark-examples.jar 1
Pi is roughly 3.137151371513715

 

I'm now trying to use rest api to submit the job from same edge node but job getting stuck at ACCEPTED state for long time. After verifying yarn node manager logs found this error:

 

org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS];

 

Submitted commands:

 

-sh-4.1$ curl --proxy "" --negotiate -u: -s -i -k -X POST https://RM_HOST:8090/ws/v1/cluster/apps/new-application

 

Generated new application-id successfully

 

-sh-4.1$ curl --proxy "" --negotiate -u: -s -i -k -v -X POST -d @1.json -H "Content-type: application/json" 'https://RM_HOST:8090/ws/v1/cluster/apps'

 

I can see job getting triggered and going to ACCEPTED state, after 20 mins job FAILED.  I verified yarn_nodemanager_resource_memory_mb property and it set to 8GB. Do I need to provide --keytab and --principal parameters here in json file? 

 

1.json file contents:

{
"application-id":"application_1526580785056_1981",
"application-name":"test-rest-api",
"application-type":"SPARK",
"am-container-spec": {
"commands":{
"command":"spark-submit --class org.apache.spark.examples.SparkPi --deploy-mode cluster --master yarn /opt/cloudera/parcels/CDH/lib/spark/lib/spark-examples.jar 1"
}
}
}

 

Appreciate your help.

 

Thank you