Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Execution of hadoop-mapreduce-examples.jar using the Yarn REST API

Solved Go to solution

Execution of hadoop-mapreduce-examples.jar using the Yarn REST API

New Contributor

Hello,

I'm trying to execute some existing examples using the Rest API (with or without using the Knox gateway)

It seems to work, but the task is always marked as failed in the Yarn Web UI.

13918-queue.png

I Use the hadoop-mapreduce-examples.jar to launch a wordcount example. It creates a sub task which is properly executed, it creates all the expected files in /user/yarn/output10.

But the parent task always failed with an exit code = 0 ( looks like a success , but it's not)

13920-appli.png

I send a POST request on this URL (I did the new-application command before to get my application-id)

https://guest:guest-password@serverip:8443/gateway/default/resourcemanager/v1/cluster/apps

(I tried also without knox : http://serverip:8088/ws/v1/cluster/apps, with the same result)

And I transmit this JSON command :

{

"application-id":"application_1490174038938_0018",

"application-name":"WordCount",

"am-container-spec":

{

"commands": {

"command":"yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar wordcount /user/yarn/input /user/yarn/output10"

}

},

"max-app-attempts":1,

"application-type":"YARN"

}

I tried a lot of other JSON config with the same result (this one is the simpliest I tried)

Do you have any idea what I did wrong ?

Thanks in advance

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Execution of hadoop-mapreduce-examples.jar using the Yarn REST API

You can try WebHCat, its mapreduce command

6 REPLIES 6

Re: Execution of hadoop-mapreduce-examples.jar using the Yarn REST API

New Contributor

I forgot this :

Ambari version : Version 2.4.2.0

Hadoop version 2.5.0.0-1245 installed from Ambari (not a sandbox)

Highlighted

Re: Execution of hadoop-mapreduce-examples.jar using the Yarn REST API

Hi @dvt isoft,

Can you please post the application logs and RM logs to debug further into this.

Re: Execution of hadoop-mapreduce-examples.jar using the Yarn REST API

New Contributor

Re: Execution of hadoop-mapreduce-examples.jar using the Yarn REST API

Hi @dvt isoft,

You cannt actually submit MR / Tez job via Yarn Webservices. Can you please try with Distributed Shell application if you want to use REST api? Or you can store your job in HDFS and use OOZIE workflow over it or use SPARK.

https://community.hortonworks.com/questions/89692/can-we-submit-a-yarn-job-through-rest.html

Re: Execution of hadoop-mapreduce-examples.jar using the Yarn REST API

You can try WebHCat, its mapreduce command

Re: Execution of hadoop-mapreduce-examples.jar using the Yarn REST API

New Contributor

Hi @Predrag Minovic and @ssathish,

I tried with WebHCat, it works.

I already tried with the Distributed Shell application, but I didn't success. Maybe I did wrong.

But I think I will try with Oozie soon.

Thanks for you help