question How to write rest api for spark job? in Archives of Support Questions (Read Only)

How to write rest api for spark job?

Manus — Wed, 21 Sep 2016 09:52:29 GMT

Hi guys,

I have written spark job which read csv file and on top of csv data am calling dataframe api to extract the profile information.

I am manually submiting spark job by using spark-submit command but I want to write jersey rest api as wrapper on top of spark job and which is i have done but am not getting

How I should launch my spark job from rest service method?

I am giving call to rest api using tomcat,so is it possible to launch spark job in tomcat web application? If not then,

Is there any solution to achieve this functionality?

Thanks in advance.

Re: How to write rest api for spark job?

rajkumar_singh — Wed, 21 Sep 2016 09:56:03 GMT

you can use spark job rest server which will allow you to submit the job using rest api.. please follow https://github.com/spark-jobserver/spark-jobserver

Re: How to write rest api for spark job?

Manus — Wed, 21 Sep 2016 10:06:10 GMT

Thanks Rajkumar,

Do we need some extra configuration to have spark job server?

If yes,then what are those steps?

Can we pass command line option to curl call while submitting spark job?

Re: How to write rest api for spark job?

rajkumar_singh — Wed, 21 Sep 2016 10:25:37 GMT

you dont need any configuration params and you can submit spark job using curl commands