question Re: /usr/lib/hadoop-mapreduce/hadoop-streamingxxxx.jar not found in Archives of Support Questions (Read Only)

/usr/lib/hadoop-mapreduce/hadoop-streamingxxxx.jar not found

zaher_mahdhi — Tue, 01 Mar 2016 06:18:29 GMT

Hi all,

I was following this tutorial:

http://hortonworks.com/blog/using-r-and-other-non-java-languages-in-mapreduce-and-hive/

and I couldn't find hadoop-streamingxxxx.jar. I'm using a cluster with hdp-2.3.4.0-3485.

Does any know where to find it or how to add it ?

Thanks 🙂

Re: /usr/lib/hadoop-mapreduce/hadoop-streamingxxxx.jar not found

aervits — Tue, 01 Mar 2016 07:16:57 GMT

@Zaher Mahdhi

you can find streaming jar in /usr/hdp/current/hadoop-mapreduce-client, make sure mapreduce, hdfs and yarn clients are installed on your machine.

[root@sandbox hadoop-mapreduce-client]# ls hadoop-streaming*
hadoop-streaming-2.7.1.2.3.2.0-2950.jar  hadoop-streaming.jar
[root@sandbox hadoop-mapreduce-client]# pwd
/usr/hdp/current/hadoop-mapreduce-client

Re: /usr/lib/hadoop-mapreduce/hadoop-streamingxxxx.jar not found

aervits — Tue, 01 Mar 2016 07:29:45 GMT

You can install it using Ambari, go to the host page and add mapreduce client, doesn't hurt to add yarn and hdfs client also

Re: /usr/lib/hadoop-mapreduce/hadoop-streamingxxxx.jar not found

nsabharwal — Tue, 01 Mar 2016 08:26:55 GMT

@Zaher Mahdhi

I see that you are using HDP 2.3.4.0-3485

See the following

[root@phdns01 ~]# find / -name hadoop-streaming*

/usr/hdp/2.3.4.0-3485/hadoop-mapreduce/hadoop-streaming-2.7.1.2.3.4.0-3485.jar

/usr/hdp/2.3.4.0-3485/hadoop-mapreduce/hadoop-streaming.jar

/usr/hdp/2.3.4.0-3485/oozie/share/lib/mapreduce-streaming/hadoop-streaming-2.7.1.2.3.4.0-3485.jar

[root@phdns01 ~]#

Re: /usr/lib/hadoop-mapreduce/hadoop-streamingxxxx.jar not found

zaher_mahdhi — Tue, 01 Mar 2016 08:57:55 GMT

@Artem Ervits thanks 🙂

Re: /usr/lib/hadoop-mapreduce/hadoop-streamingxxxx.jar not found

zaher_mahdhi — Tue, 01 Mar 2016 08:59:27 GMT

@Neeraj Sabharwal Thank u for your answer

Re: /usr/lib/hadoop-mapreduce/hadoop-streamingxxxx.jar not found

zaher_mahdhi — Tue, 01 Mar 2016 09:01:09 GMT

When testing the example I got this error 😕 :

16/03/01 01:57:29 INFO mapreduce.Job: Task Id : attempt_1456773989186_0006_r_000001_2, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
    at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
    at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:134)
    at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244)
    at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

Re: /usr/lib/hadoop-mapreduce/hadoop-streamingxxxx.jar not found

nsabharwal — Tue, 01 Mar 2016 09:07:38 GMT

@Zaher Mahdhi Open this as new question to avoid confusion

Re: /usr/lib/hadoop-mapreduce/hadoop-streamingxxxx.jar not found

aervits — Tue, 01 Mar 2016 09:37:31 GMT

@Zaher Mahdhi I agree, can you please post this as a new question and provide steps to reproduce the problem.