Support Questions
Find answers, ask questions, and share your expertise

HDPCD: Apache Spark certification, submitting job via scala....what tools?

HDPCD: Apache Spark certification, submitting job via scala....what tools?

New Contributor

Hi all,

one of the exam objectives of the HDPCD Apache Spark certification is submitting the job to YARN....But to do this we need to build the fat jar, yes I work with Scala...Can someone please clarify what tools during exam we're going to have to build the jar? Are we going to have prepared sbt.build file or is there going to be an empty maven project with dependencies....It's not like with python where you can submit py file....

Thank you very much in advance....exam objective regarding this is very blurry....

best regards

Tomas

3 REPLIES 3

Re: HDPCD: Apache Spark certification, submitting job via scala....what tools?

Explorer

During the exam, if you have a question about this, you will be asked to submit a spark application, and in the context of the question, you will be informed the JAR name, the JAR main class and how the application should be processed (local, etc.).

I recently completed certification and it was so.

Hope to help you.

Re: HDPCD: Apache Spark certification, submitting job via scala....what tools?

New Contributor

@Gilvan can you please post an example of the code how to run a jar file on yarn. I am confused, in the question will they give me the class name and path to the jar file. my understanding is I have run below script

spark-submit --master yarn --class "whats the class name here" /user/jarfilename.jar

do i also need to type ./bin/spark-submit

Re: HDPCD: Apache Spark certification, submitting job via scala....what tools?

Explorer

Hi @Vinod Kumar, you're right.

If this type of question is part of your proof the answer will be as you quoted, via spark-submit informing the parameters required for submission of the job.