Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Execution of hadoop-mapreduce-examples.jar using the Yarn REST API

avatar
Contributor

Hello,

I'm trying to execute some existing examples using the Rest API (with or without using the Knox gateway)

It seems to work, but the task is always marked as failed in the Yarn Web UI.

13918-queue.png

I Use the hadoop-mapreduce-examples.jar to launch a wordcount example. It creates a sub task which is properly executed, it creates all the expected files in /user/yarn/output10.

But the parent task always failed with an exit code = 0 ( looks like a success 😞 , but it's not)

13920-appli.png

I send a POST request on this URL (I did the new-application command before to get my application-id)

https://guest:guest-password@serverip:8443/gateway/default/resourcemanager/v1/cluster/apps

(I tried also without knox : http://serverip:8088/ws/v1/cluster/apps, with the same result)

And I transmit this JSON command :

{

"application-id":"application_1490174038938_0018",

"application-name":"WordCount",

"am-container-spec":

{

"commands": {

"command":"yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar wordcount /user/yarn/input /user/yarn/output10"

}

},

"max-app-attempts":1,

"application-type":"YARN"

}

I tried a lot of other JSON config with the same result (this one is the simpliest I tried)

Do you have any idea what I did wrong ?

Thanks in advance

1 ACCEPTED SOLUTION

avatar
Master Guru
6 REPLIES 6

avatar
Contributor

I forgot this :

Ambari version : Version 2.4.2.0

Hadoop version 2.5.0.0-1245 installed from Ambari (not a sandbox)

avatar

Hi @dvt isoft,

Can you please post the application logs and RM logs to debug further into this.

avatar
Contributor

Thanks @ssathish,

here are my log files.

avatar

Hi @dvt isoft,

You cannt actually submit MR / Tez job via Yarn Webservices. Can you please try with Distributed Shell application if you want to use REST api? Or you can store your job in HDFS and use OOZIE workflow over it or use SPARK.

https://community.hortonworks.com/questions/89692/can-we-submit-a-yarn-job-through-rest.html

avatar
Master Guru

You can try WebHCat, its mapreduce command

avatar
Contributor

Hi @Predrag Minovic and @ssathish,

I tried with WebHCat, it works.

I already tried with the Distributed Shell application, but I didn't success. Maybe I did wrong.

But I think I will try with Oozie soon.

Thanks for you help