question Re: Execution of hadoop-mapreduce-examples.jar using the Yarn REST API in Archives of Support Questions (Read Only)

Execution of hadoop-mapreduce-examples.jar using the Yarn REST API

dvt_isoft — Sun, 18 Aug 2019 10:42:49 GMT

Hello,

I'm trying to execute some existing examples using the Rest API (with or without using the Knox gateway)

It seems to work, but the task is always marked as failed in the Yarn Web UI.

I Use the hadoop-mapreduce-examples.jar to launch a wordcount example. It creates a sub task which is properly executed, it creates all the expected files in /user/yarn/output10.

But the parent task always failed with an exit code = 0 ( looks like a success 😞 , but it's not)

I send a POST request on this URL (I did the new-application command before to get my application-id)

https://guest:guest-password@serverip:8443/gateway/default/resourcemanager/v1/cluster/apps

(I tried also without knox : http://serverip:8088/ws/v1/cluster/apps, with the same result)

And I transmit this JSON command :

{

"application-id":"application_1490174038938_0018",

"application-name":"WordCount",

"am-container-spec":

{

"commands": {

"command":"yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar wordcount /user/yarn/input /user/yarn/output10"

}

"max-app-attempts":1,

"application-type":"YARN"

}

I tried a lot of other JSON config with the same result (this one is the simpliest I tried)

Do you have any idea what I did wrong ?

Thanks in advance

Re: Execution of hadoop-mapreduce-examples.jar using the Yarn REST API

dvt_isoft — Thu, 23 Mar 2017 21:37:24 GMT

I forgot this :

Ambari version : Version 2.4.2.0

Hadoop version 2.5.0.0-1245 installed from Ambari (not a sandbox)

Re: Execution of hadoop-mapreduce-examples.jar using the Yarn REST API

ssathish — Sat, 25 Mar 2017 06:52:56 GMT

Hi @dvt isoft,

Can you please post the application logs and RM logs to debug further into this.

Re: Execution of hadoop-mapreduce-examples.jar using the Yarn REST API

dvt_isoft — Mon, 27 Mar 2017 15:15:22 GMT

Thanks @ssathish,

here are my log files.

Re: Execution of hadoop-mapreduce-examples.jar using the Yarn REST API

ssathish — Thu, 30 Mar 2017 05:25:44 GMT

Hi @dvt isoft,

You cannt actually submit MR / Tez job via Yarn Webservices. Can you please try with Distributed Shell application if you want to use REST api? Or you can store your job in HDFS and use OOZIE workflow over it or use SPARK.

https://community.hortonworks.com/questions/89692/can-we-submit-a-yarn-job-through-rest.html

Re: Execution of hadoop-mapreduce-examples.jar using the Yarn REST API

pminovic — Thu, 30 Mar 2017 06:24:46 GMT

You can try WebHCat, its mapreduce command

Re: Execution of hadoop-mapreduce-examples.jar using the Yarn REST API

dvt_isoft — Thu, 30 Mar 2017 18:49:12 GMT

Hi @Predrag Minovic and @ssathish,

I tried with WebHCat, it works.

I already tried with the Distributed Shell application, but I didn't success. Maybe I did wrong.

But I think I will try with Oozie soon.

Thanks for you help