Reply
Explorer
Posts: 12
Registered: ‎08-20-2015

Yarn RM REST api to submit map-reduce job

Hi - I am looking for some information about can we submit mapreduce job using Yarn RM REST Apis. I have a mapreduce jar which works well with hadoop jar commond unix shell. I want to submit the same job remotly via my web application. I am thinking to use Yarn RM REST Apis. Specially Cluster Applications API(Submit Application)

 

My Questions

1. Is this possible ? Is this the right approch.

2. Is there any sample code avaliable where I can see how to submit the MR job via REST Api. The documentation shows example for ShellCommond only which is not very clear to me.

 

Thanks in advance.

MG

 

Posts: 1,896
Kudos: 433
Solutions: 303
Registered: ‎07-31-2013

Re: Yarn RM REST api to submit map-reduce job

While it is possible to submit an application via the REST API of RM's
WebServices, its not a trivial thing to setup the same for a complex
application such as MapReduce2 (which will make use of every available API
hook).

You may find it easier instead to store your job on HDFS, and use an Oozie
workflow over it. Oozie supports a REST API that is far easier to use to
fire and control jobs:
http://archive.cloudera.com/cdh5/cdh/5/oozie/WebServicesAPI.html#Job_Submission
Highlighted
Explorer
Posts: 12
Registered: ‎08-20-2015

Re: Yarn RM REST api to submit map-reduce job

Thanks for the reply. We are moving on oozie wf webservice. I agree that is very easy as compare to yarn REST apis. Thanks.
Announcements

Our community is getting a little larger. And a lot better.


Learn More about the Cloudera and Hortonworks community merger planned for late July and early August.