Execution of hadoop-mapreduce-examples.jar using the Yarn REST API

[#1]

Hello,

I'm trying to execute some existing examples using the Rest API (with or without using the Knox gateway)

It seems to work, but the task is always marked as failed in the Yarn Web UI.

I Use the hadoop-mapreduce-examples.jar to launch a wordcount example. It creates a sub task which is properly executed, it creates all the expected files in /user/yarn/output10.

But the parent task always failed with an exit code = 0 ( looks like a success 😞 , but it's not)

I send a POST request on this URL (I did the new-application command before to get my application-id)

https://guest:guest-password@serverip:8443/gateway/default/resourcemanager/v1/cluster/apps

(I tried also without knox : http://serverip:8088/ws/v1/cluster/apps, with the same result)

And I transmit this JSON command :

{

"application-id":"application_1490174038938_0018",

"application-name":"WordCount",

"am-container-spec":

{

"commands": {

"command":"yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar wordcount /user/yarn/input /user/yarn/output10"

}

},

"max-app-attempts":1,

"application-type":"YARN"

}

I tried a lot of other JSON config with the same result (this one is the simpliest I tried)

Do you have any idea what I did wrong ?

Thanks in advance

[#2]

I forgot this :

Ambari version : Version 2.4.2.0

Hadoop version 2.5.0.0-1245 installed from Ambari (not a sandbox)

[#3]

Hi @dvt isoft,

Can you please post the application logs and RM logs to debug further into this.

[#4]

Thanks @ssathish,

here are my log files.

[#5]

Hi @dvt isoft,

You cannt actually submit MR / Tez job via Yarn Webservices. Can you please try with Distributed Shell application if you want to use REST api? Or you can store your job in HDFS and use OOZIE workflow over it or use SPARK.

https://community.hortonworks.com/questions/89692/can-we-submit-a-yarn-job-through-rest.html

[#6 · Accepted solution]

You can try WebHCat, its mapreduce command

[#7]

Hi @Predrag Minovic and @ssathish,

I tried with WebHCat, it works.

I already tried with the Distributed Shell application, but I didn't success. Maybe I did wrong.

But I think I will try with Oozie soon.

Thanks for you help