Support Questions

Find answers, ask questions, and share your expertise

How to submit spark jobs from a remote machine?

avatar

Steps to be performed before submiting

the spark jobs from a remote machine. I have tried adding the core.xml and yarn.xml file form master to the remote machine in the spark conf foldern. Still its not working. Can anyone suggest how to perform this step.

1 REPLY 1

avatar
@Deepak Nayak

We can submit the spark jobs on remote cluster using the livy server using Rest calls. Below are the couple of links for examples:

https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-apache-spark-livy-rest-interface

https://github.com/cloudera/livy

Hope this helps you.